Numerical approaches for free-form lensing: area parameterization free-form lensing

ABSTRACT

A free-form lens (for example a phase modulator, lens or deformable mirror) may be made to reproduce a light pattern specified by image data. Source regions on the free-form lens are mapped to target regions areas on an image. Areas of the source regions are adjusted to vary the amount of light delivered to each of the target regions. Adjustment of the source areas may be achieved using a L-BFGS optimization which preferably incorporates smoothness and curl regularizers. Embodiments apply parallel processing to obtain control values for a free form lens in real time or near real time. Apparatus may process image data and display an image by controlling a dynamically variable free form lens using the processed image data.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.15/406,942 filed 16 Jan. 2017. U.S. application Ser. No. 15/406,942 is acontinuation of PCT International Application No. PCT/CA2015/050730which is hereby incorporated herein by reference for all purposes. PCTInternational Application No. PCT/CA2015/050730 claims priority fromU.S. Application No. 62/031,250 filed 31 Jul. 2014 and U.S. ApplicationNo. 62/194,728 filed 20 Jul. 2015. This application claims the benefitunder 35 U.S.C. § 119 of U.S. Application No. 62/031,250 filed 31 Jul.2014 and entitled METHODS AND APPARATUS LIGHT STEERING USINGPHASE-MODULATED IMAGING and U.S. Application No. 62/194,728 filed 20Jul. 2015 and entitled NUMERICAL APPROACHES FOR FREE-FORM LENSING: AREAPARAMETERIZATION FREE-FORM LENSING, both of which are herebyincorporated herein by reference for all purposes.

FIELD

This invention relates to projecting light using free-form lenses. Insome embodiments the free form lenses comprise spatial phase modulators.Embodiments provide light projectors, methods for projecting light,components for light projectors and tangible media containingmachine-readable instructions for implementing described methods.

BACKGROUND

There are many applications in which it is desirable to project patternsof light. These include displays (e.g. cinema projectors, computerdisplays, televisions, advertising displays—e.g. billboards, virtualreality displays etc.) as well as architectural lighting, automobilelighting (e.g. headlights, driving lights) and special effects lighting(e.g. theater stage lighting, concert lighting).

One technical problem is to provide displays capable of achieving highluminance levels. High luminance levels may be used to project lightpatterns having high dynamic ranges and/or to project light patternsviewable under various ambient lighting conditions, for example. Withmany current display technologies achieving high luminance levels isaccompanied by undesirably high power consumption.

A major motivation for using light-steering in an imaging system is thatpeak luminance levels far above full-screen white (FSW) can be achieved.This is possible as light taken from the dark areas can be redistributed(steered) to areas that require higher luminance. Another consequence ofsteering light is that deeper black levels can also be reached. Byextending the highlights and black levels in an image, a wider range oflight levels (“increased contrast”) can be displayed simultaneously.

Light can be steered by free-form lensing. Determining a configurationfor a free-form lens that will steer light to provide a desired lightpattern is computationally difficult for all but very simple lightpatterns. Computational caustics is a field of study which relates tohow refractive and/or reflective optical layers affect distribution oflight.

Some approaches to computational caustics involve determining anarrangement of pre-specified discrete primitives such as planar,quadratic or Gaussian patches. Methods based on pre-specified primitivesoften suffer from edge effects when primitives do not meet in acompatible way.

Some alternative approaches apply optimal transportation. Optimaltransportation seeks a mapping from a source to a target distributionsuch that a user-specified cost-function is minimized. Optimaltransportation has been applied in areas as diverse as operationsresearch and mesh processing: an optimal transport formulation is usedto determine a mapping of a source intensity distribution at the lensplane to a target distribution at the image plane. This approach canachieve high-contrast and very good image quality, but comes withhigh-computational cost. Typical images may require hours ofcomputation. Furthermore the computation is difficult to parallelize.

There remains a need for light projectors which can create desired lightfields. There is a particular need for ways to generate desired lightfields that are computationally efficient and yet provide qualityreproduction of a desired light field. There is also a desire formethods and apparatus for reproducing light fields that are energyefficient.

SUMMARY

This invention has a number of aspects. Some aspects provide lightprojectors. Some aspects provide methods for generating free-form optics(which may comprise spatial phase modulators) corresponding to desiredlight fields (which may comprise images—the images may be video framesin some embodiments). Some aspects provide methods for processing dataspecifying a light field to yield a configuration for a correspondingfree-form lens.

This invention also relates to free-form lensing. Free-form lensinginvolves generating a desired light field by redistributing light from asource using a customized optical layer. Embodiments of the inventionprovide light projectors comprising free-form lenses, methods forprojecting specified light fields, and methods and apparatus forprocessing data defining desired light patterns to generateconfigurations for free form lenses. In example embodiments the opticallayer comprises a customized refractive and/or reflective element or aphase modulator. “Computational caustics” is a related field.

One example aspect provides methods for controlling a phase modulator todisplay an image defined by image data. The method comprises defining aplurality of non-overlapping source regions on a two-dimensional phasemodulator and a plurality of display regions at a display plane, each ofthe source regions having a boundary and a source area and beingassociated with a corresponding one of the display regions; each of thedisplay regions having a corresponding display region area; based on theimage data, assigning a target light intensity value to each of aplurality of the display regions; adjusting: a configuration for thesource regions; or a configuration for the display regions; orconfigurations for both the source regions and the display regions suchthat ratios of the display areas of the display regions to the sourceareas of the corresponding source regions is proportional to a ratio ofsource light intensity values for the source regions to the target lightintensity value assigned to the corresponding display region; generatinga phase surface for each of the source areas, the phase surfaceconfigured to redirect light incident on the source area onto thecorresponding display area; and controlling the phase modulator toprovide the phase surfaces for the source regions and illuminating thesource regions with incident light according to the source intensityvalues.

Another example aspect provides a method for generating a free form lensconfiguration useful for displaying an image defined by image data. Themethod comprises: providing a model of a two-dimensional light sourcecomprising a plurality of non-overlapping source regions. Each of thesource regions has a boundary, a corresponding source light intensityvalue and a source area. Each of the source regions is associated with acorresponding display region of a display. Each of the display regionshas a target area. The method proceeds to assign a light intensity valueto each of the display regions based on the image data. The method setsa target source area for each of the source regions such that a ratio ofthe target source area of the source region to the display area of thecorresponding display region is proportional to a ratio of the lightintensity value assigned to the corresponding display region to thesource light intensity value for the source region. The method performsan optimization to determine configurations for the boundaries of thesource regions which best satisfy an objective function which quantifiesaggregate deviations of the areas of the source regions from the targetsource areas corresponding to the source regions. Based on theconfigurations of the source region boundaries after the optimizationthe method determines a normal vector for each of the source regions andintegrates the normal vectors to yield a solution phase functionrelating a phase to position in two dimensions. Where a phase modulatoris used to provide the free-form lens the solution phase function may beapplied to drive the phase modulator.

In some embodiments the source regions comprise non-overlapping sourcetiles defined by lines extending between a plurality of source vertices.Each of the source vertices has a location. In some embodiments thedisplay tiles are defined by lines extending between a plurality ofdisplay vertices.

In some embodiments the source tiles and display tiles are triangles.The optimization may determine optimized locations for the sourcevertices.

In some embodiments determining the normal vectors for the sourcevertices is based on in-plane displacements of the source verticesrelative to corresponding ones of the display vertices.

In some embodiments optimizing comprises applying a limited memoryBroyden-Fletcher-Goldfarb-Shanno algorithm. Some embodiments compriseperforming the optimization in a series of iterations at progressivelyfiner scales such that, in each iteration the number of source verticesand display vertices is increased and the vertex positions for animmediately previous iteration are used as starting configurations for acurrent iteration.

Further aspects and example embodiments are illustrated in theaccompanying drawings and/or described in the following description.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings illustrate non-limiting example embodiments ofthe invention.

FIG. 1 is a schematic illustration of an example light projector. Theincoming known light distribution (which may be but is not necessarilyuniform) is first steered, then amplitude-modulated. Amplitudemodulation may be provided by a spatial light modulator. The embodimentof FIG. 1 illustrates a projector that uses transmissive elements.Implementations which apply other (e.g. reflective) light steeringelements and/or amplitude modulators are possible.

FIGS. 2A and 2B illustrate schematically a free space implementation.Light falls onto a spatial phase modulator (e.g. a HOLOEYE™ LETO seriesphase only modulator is used in some embodiments) with a knowndistribution (which may be uniform). After phase modulation the lightcontinues to a spatial light modulator (SLM). In one implementation, thedifferent regions on the spatial phase modulator may be of differentsizes, whereas the regions on the SLM may be of the same size. Thisenables altering of the light intensity in each respective region of theSLM. Intensity distributions for each step in the light path isindicated in the diagram. The spatial phase modulator and the SLM appearas being the same size in this diagram, but that is not a requirement.

FIGS. 3A and 3B illustrate an example implementation in whichintegration rods are used. The steered light from each region of thespatial phase modulator is focused onto centers of integration rods inan array of integration rods. Example intensity distributions for eachstep in the light path is indicated in the diagram. This light isrelayed onto the SLM for a final clean-up. The factor α is a consequenceof the focusing of the light from the spatial phase modulator onto eachintegration rod and should approximately account for power conservation.

FIG. 4 is a diagram that illustrates the flow of an example shift andscale SNS algorithm and its incorporation with a physical system. Thetarget image is bisected and in this example the required intensities inthe two halves are found to be 15 (left) and 5 right) (arbitraryluminance units). This leads us to split the spatial phase modulatorinto two regions, where the area of the left hand side is 3× that of theright hand side. The light incident on the spatial phase modulator isthen steered onto two equi-sized regions on the SLM.

FIG. 5 shows an example of a tilted, parabolic lens. A region of aspatial phase modulator may be configured to provide such a lens. Theconfiguration of the lens may be controlled depending on the locationand size of a corresponding region on the SLM to which the lens shouldsteer light.

FIG. 6A is a diagram (not to scale) suggesting shapes of lenses that maybe implemented on a spatial phase modulator for a free spaceimplementation. The sizes of the spatial phase modulator regions may bedetermined by the SNS algorithm. The focal points of each region of thespatial phase modulator are indicated in the diagram, to the right ofthe LETO-SLM assembly for region 1, and to the left of the LETO-SLMassembly for region 2.

FIG. 6B is a diagram (not to scale) suggesting a shape of lens that maybe implemented on a spatial phase modulator (e.g. LETO) for anintegration rod implementation. The sizes of the regions on the spatiallight modulator may be determined by the SNS algorithm. Diagram is notto scale.

FIGS. 7A, 7B and 7C illustrate processing image data to determinedesired luminance levels for different display regions. An 8×16sectioned image of Marilyn. (FIG. 7A) shows the complete image with the8×16 zones superimposed. FIG. 7B shows the first bisection in x withresulting mean luminance levels. FIG. 7C shows the second bisection in yfor each of the two halves calculated and shown in FIG. 7B.

FIG. 8 illustrates an 8×16 zone set of lenses calculated for the imageof Marilyn shown in FIG. 7A. The units on the right hand side are in mm.

FIG. 9 shows the lens calculated in FIG. 8, wrapped to multiples of thewavelength of the light, lambda (638 nm in this particular example). Theunits on the right hand side (RHS) are in multiples of lambda. Thismathematical mod-operation, mod(phase-pattern, lambda), is also known asphase-wrapping.

FIG. 10 is a calculated (ideal) output of the SNS-derived lens.

FIG. 11 shows how light from different regions or segments in themodulator plane (these may be called source regions) is redistributed byscaling and shifting it towards corresponding regions or segments in thetarget plane (these may be called display regions).

FIG. 12 is a diagram showing a modulator plane and target image plane,with corresponding regions on each, as well as a point array to be usedin the mathematical derivation. In some embodiments the point array isconstructed so that each point corresponds to a pixel of a spatial phasemodulator in the modulator plane.

FIG. 13 is a diagram illustrating optical path lengths between amodulator plane point array and the corresponding target image planepoint array.

FIG. 14 is a diagram illustrating optical path lengths between amodulator plane point array and the corresponding target image planepoint array according to an embodiment where the path length profile ismade up of distances separating the points in a source region frompoints in a virtual parabola associated with a corresponding displayregion.

FIG. 15 is a diagram of an example physical lens. Light enters along theoptical axis, is transmitted without deflection after entering the backlens surface and is then refracted at the front lens surface which makesan angle θ₁ with respect to the optical axis. The transmitted angleθ_(t) with respect to the optical axis is then given by Snell's law.

FIGS. 16A through 16C are a set of images that illustrate the effect ofpadding. Images that are not padded properly often have boundarydistortions due to periodicity assumptions in the Fourier transform.Mirrored padding around the target results in a periodic image. Thisreduces boundary distortions but reduces contrast.

FIGS. 17A to 17D illustrate the effect of varying a smoothness parameteron image quality. Reducing the smoothness parameter can result insignificantly increased contrast but can also result in perceptiblecaustic artefacts.

FIGS. 18A through 18D illustrate the effect of regularization. FIGS. 18Aand 18B are computed point positions for area-based parameterizationswith and without curl-regularization (with weight 1.0). FIGS. 18C and18D are the resulting output images. Incorporating curl-regularizationhelps to reduce shearing distortions and results in displacements.

FIG. 19A illustrates an example mapping of point positions for theMarilyn image. Marilyn's face and hair is mapped to nearly the entirelens surface, dramatically compressing low-intensity regions. Despitehigh-compression, the vast majority of mapped quadrilaterals are convex,indicating a bijective parameterization. Local contrast in the resultingimage is determined by the ratio of areas of adjacent quadrilaterals.FIG. 19B is a magnified part of FIG. 19A corresponding to Marilyn's eye.

FIGS. 20A through 20C are images of Einstein which compare Fourierparaxial (FIG. 20A) and area-parameterization approaches (FIG. 20B) tofree-form lensing. The area-parameterization image uses a gamma exponentof 3.0. FIG. 20C is the target image.

FIGS. 21A through 21C are images comparing Fourier paraxial (FIG. 21A)and area-parameterization (FIG. 21B) approaches on the fram-ref image.FIG. 21C is the target.

FIGS. 22A through 22C are images comparing Fourier paraxial (FIG. 22A)and area-parameterization (FIG. 22B) approaches on the Lena image. FIG.22C is the target.

FIGS. 23A through 23C are images comparing Fourier paraxial (FIG. 23A)and area-parameterization (FIG. 23B) approaches on the Marilyn image.FIG. 23C is the target.

FIGS. 24A through 24C are images comparing Fourier paraxial (FIG. 24A)and area-parameterization (FIG. 24B) approaches on the “trooper” image.FIG. 24C is the target.

FIGS. 25A to 25D illustrate the effect of scale of area parameterizationon the Marilyn image shown in FIG. 25E. Increasing resolution reducesthe artefacts at highly stretched regions, indicating that a spatiallyadaptive discretization could be beneficial.

FIGS. 26A to 26D illustrate the effect of curl regularization on areaparameterization results for the Einstein image shown in FIG. 26E.Increasing weight for the curl-regularizer results in more integrabledisplacements which reduces stretching and shearing artefacts, butdecreases contrast. A typical value is 10.0.

FIGS. 27A to 27D illustrate the effect of varying a smoothness parameteron area parameterization results for the Marilyn image shown in FIG.27E. Low values for the smoothness parameter result in higher-contrast,but more pronounced artefacts. High values for the smoothness parameterreduce contrast but help to suppress artefacts. A typical value is 0.05.

FIGS. 28A to 28D illustrate the effect of varying a minimum area on areaparameterization results for the Einstein image shown in FIG. 28E. Thisparameter acts as a hard floor on the minimum area targeted by theoptimization. When set too low, it results in low-quality images, butexcellent contrast. When set too high, it prevents effective lightredistribution. A typical value is 0.05.

FIGS. 29B and 29C are respectively an area parameterization image:in-scene contrast 106:1, peak brightness 2.8×FSW and a paraxialdeblurring image: in-scene contrast: 67:1, peak brightness: 2:9×FSW forthe Lena image of FIG. 29A.

FIGS. 30B and 30C are respectively an area-parameterization image:in-scene contrast 582:1, peak brightness 11.92×FSW and a paraxialdeblurring image: in-scene contrast: 173:1, peak brightness 10.0×FSW forthe Marilyn image of FIG. 30A.

FIGS. 31B and 31C are respectively an area-parameterization image:in-scene contrast 377:1, peak brightness 6.2×FSW, and a paraxialdeblurring image: in-scene contrast 101:1, peak brightness 4×FSW for the“fram-ref” image shown in FIG. 31A.

FIGS. 32B and 32C are respectively an area-parameterization image:in-scene contrast 759:1, peak brightness 13.15×FSW, and a paraxialdeblurring image: in-scene contrast 104:1, peak brightness 8.1×FSW forthe Einstein image shown in FIG. 32A.

FIGS. 33A to 33H are photographs of projections from a prototypeprojector (LETO) with broadband illumination comparingarea-parameterization and paraxial deblurring methods with same camerasettings.

FIGS. 34A to 34D are photographs of projections from a prototypeprojector (LETO) with broadband illumination.

FIG. 35B is an experimental capture of the “avengers” image of FIG. 35A:in-scene contrast 1025:1, peak brightness 8.84×FSW.

FIG. 36B is an experimental capture of the “candle” image of FIG. 36A:in-scene contrast 697:1, peak brightness 9.85×FSW.

FIG. 37B is an experimental capture of the “F1” image of FIG. 37A:in-scene contrast 301:1, peak brightness 6.18×FSW.

FIG. 38B is an experimental capture of the “clouds” image of FIG. 38A:in-scene contrast 697:1, peak brightness 7.42×FSW.

FIG. 39B is an experimental capture of the “space” image of FIG. 39A,in-scene contrast 935:1, peak brightness 16.2×FSW.

FIG. 40 schematically illustrates a mapping between source regions andtarget regions.

FIG. 41 is a block diagram illustrating apparatus according to anexample embodiment.

FIG. 42 is a block diagram illustrating a projector according to anexample embodiment.

FIGS. 43A and 43B are a flow chart illustrating a method according to anexample embodiment.

DETAILED DESCRIPTION

Throughout the following description, specific details are set forth inorder to provide a more thorough understanding of the invention.However, the invention may be practiced without these particulars. Inother instances, well known elements have not been shown or described indetail to avoid unnecessarily obscuring the invention. Accordingly, thespecification and drawings are to be regarded in an illustrative, ratherthan a restrictive sense.

This document describes various embodiments of light projector as wellas methods for configuring free form lenses to project desired lightpatterns. Some embodiments combine a light steering stage comprising afree form lens (which is provided by a spatial phase modulator in someembodiments) with a spatial amplitude modulation phase.

In some embodiments a configuration for the light steering phase isarrived at by a method that comprises associating source regions at afree-form lens with display regions of a projected light pattern.Desired light intensities in the display regions are adjusted by varyingthe relative areas of the display regions and their corresponding sourceregions. The relative areas of the source- and display regions may bealtered by changing areas of the source regions, changing areas of thedisplay regions or changing both areas of the source regions and areasof the display regions. The free form lens may be configured so thateach source region directs light onto the corresponding display region.In some embodiments 90% or 95% or more of the light projected by eachsource region onto the target image plane falls within the correspondingdisplay region. In some embodiments, the intensity of illumination ofthe free form lens is controlled based on an average or representativeluminance of the desired light pattern. In some embodiments spatialamplitude modulation stages are provided one or both of upstream anddownstream of the free form lens. An upstream SLM may vary the luminanceat the source regions. A downstream SLM may further modulate light thatilluminates the target image plane. The downstream SLM may have aspatial resolution finer than a resolution of the display regions insome embodiments.

The following description explains various ways to configure a free formlens in response to image data defining a desired light pattern.“Shift‘n’scale” (SNS) is a procedural, forward-only algorithm that whenused in conjunction with a phase-retarding imaging chip enables lightsteering in some embodiments. SNS beneficially can avoid or reduce edgeeffects in some embodiments. Some embodiments use computational causticsmethods that involve determining an arrangement of pre-specifieddiscrete primitives such as planar, quadratic or Gaussian patches, thatcan be used to configure a free form lens.

FIG. 1 is a basic diagram illustrating a light projector that combinesthe steering of light with amplitude modulation of the incoming lightdistribution in order to form images more efficiently.

This document describes a non-limiting example implementation of a lightprojector where light is steered by a phase-modulating imaging chip, forexample an LCoS based phase-modulating microdisplay. An example of sucha display is made by HoloEye Photonics AG called LETO (1080×1920 pixelsand about 6.4 micrometer pixel pitch). Light reflected off the LETO isincident on an amplitude modulator, in our case a Sony™ liquid crystalon silicon modulator (LCoS), which is a non-limiting example of aspatial light modulator (SLM). The image from the SLM is then relayedonto a projection screen.

Alternative implementations are possible, for example by reversing theorder of the two modulators: amplitude modulate the light first, thensteer the light. Other possible implementations include phase-modulatorsthat only modulate up to ½ of one wavelength of the light (so called“π-modulators”). Other possible amplitude modulators include the DigitalLight Projector (DLP) or Digital Micromirror Device (DMD), examples ofwhich are available from Texas Instruments.

One implementation of the system sees the incoming light distributionsteered by a LETO directly onto the SLM as indicated in FIG. 2. In otherembodiments optical systems of suitable kinds may be provided betweenthe spatial phase modulator and the SLM. Such optical systems maycomprise, for example, arrangements of one or more of lenses, mirrors,diffusers, free space, filters, etc.

As suggested in FIG. 2, steering is used to pre-modulate the lightincident on the SLM. This can be done by lensing the light from severalregions of the spatial phase modulator onto corresponding regions on theSLM. The lenses implemented are in our case simple parabolic lenses withfocal distances in x and y derived from how much amplification isrequired in the two directions. Similarly, in order to shift adistribution in the plane, a slope is applied to the lensing solution ineach region, one in x and one in y. These basic operations have led tothe name “Shift′n′scale” (SNS).

An alternative implementation of this system uses equi-sized regions onthe LETO, illuminating differently sized regions on the SLM. Thederivation for the physical model is similar to the one for thepreferred implementation.

Another alternative implementation is illustrated by FIG. 3, which showsa projector in which light is homogenized using an array of integrationrods between the spatial phase modulator and the SLM. Homogenization maybe beneficial in order to smooth out irregularities in a laser beamprofile, for example.

The output from different integration rods may have differentamplitudes, but their spatial distributions should be known orapproximately known. The focal distances for each region areapproximately the same, and indicated in FIGS. 3A and 3B. Smallvariations in the focal distances for the different regions could ensuresimilar numerical aperture or spread of light from each integration rod.The shifts for each region will vary.

Shift and Scale Algorithm

Many approaches can be used to calculate the appropriatephase-modulating image on the spatial phase modulator. In one approach,the spatial phase-modulator is divided into several regions wheredifferent lenses are defined in order to provide the required amount ofmagnification and steering for those regions. In a sense, this is likehaving a programmable array of parabolic lenses that each shifts andscales a region of light from the spatial phase modulator ontocorresponding regions on the SLM on a frame-by-frame basis. The goal ofthe SNS algorithm is to provide a fast, low-resolution version of thetarget image. If the resulting fast low resolution image does not havesufficient resolution for a particular application then one can use anamplitude-modulator to create the desired high-resolution target imageon the screen, but with minimal loss of light since excessive amplitudemodulation can be avoided.

The following two sections describe two example cases for splitting upeach of a spatial phase modulator and a target image plane (which may beon a SLM in some embodiments) into multiple regions. Alternativederivations using differently sized regions on both the spatial phasemodulator and the SLM are also possible.

Approach 1: Differently-Sized Spatial Phase Modulator Regions; EquallySized SLM Regions

The SNS algorithm analyzes the image to be displayed and effectivelytranslates intensity-requirements of the target image into arealdistributions (this is in some sense similar to the Median Cut Algorithm[Ref.: http://en.wikipedia.org/wiki/Median_cut]). SNS is a recursive,multi-scale algorithm. An example embodiments starts by comparing theright and the left side of the image and sets aside according areas onthe phase modulator to be able to match the illumination requirements ofeach side. SNS then repeatedly bisects the already-processed imageregions and again translates intensity requirements into areas. Thisprocess can be repeated recursively. A diagram outlining theregion-to-region mapping between source regions on the spatial phasemodulator (or other free form lens) and display regions on the SLM (orother target plane) after one bisection step is shown in FIG. 4.

Determining the “illumination requirements” during each bisection stepcan be done in different ways. For examples, the most stringentrequirement is that the maximum luminance levels of each part of thetarget image are to be achievable; this leaves the least amount of lightavailable for re-direction and is therefore the most conservativeapproach. Requiring only the mean luminance per region will lead to lostlight levels in each region and will surely reduce image qualityalthough this approach may be acceptable for some applications.Alternatively, one can aim to reproduce some predetermined percentage ofthe light for each region, which will require only a small amount ofper-region tone-mapping, for example by soft-clipping highlights and/orblack levels that are beyond the available dynamic range of each region.

In summary, the SNS approach uses a free-form lens such as a phasemodulator that is split into many regions whose areas may all differdepending on how much light they are required to deliver to acorresponding set of regions on the SLM. The size of the relativeregions is what determines the amount of steering and amplification perregion.

In one implementation, we determine the shape of each lens bycalculating required focal in-plane distances as well as in-plane shiftsfor each region. A simple parabolic lens can be defined as follows:lens_(i) =f _(x,i)(1−√{square root over (1−x ² /f _(x,i) ²)})+m _(x,i)x+f _(y,i)(1−√{square root over (1−y ² /f _(y,i) ²)})+m _(y,i) y  [1A]where (f_(x,i), f_(y,i)) are the focal distances in x and y of thei_(th) region, and (m_(x,i), m_(y,i)) are the tilts of the lens in thatregion. Other implementations are possible. For example, treating theincoming light distribution as bouncing off the phase modulator at aspecular angle, a surface of gradients can be derived from knowing wherethe light should be sent onto the next modulator (e.g. a SLM) or othertarget plane. This gradient map can be integrated to form a phase mapfor the spatial phase modulator.

Two example ways of relaying light from a free from lens to a SLM aredescribed above. In the “free-space approach”, the focal distance foreach region will be determined by how much magnification is requiredbetween the source region in question and the corresponding displayregion. The following expression will ensure correct magnification:f _(x,i) =D/(1−a _(x,i) /b _(x,i))  [2A]where D is the distance between the free form lens (e.g. spatial phasemodulator) and the SLM, a_(x,i) is the x-size of the source region onthe spatial phase modulator and b_(x,i) is the x-size of thecorresponding display region (e.g. on the SLM). These parameters areillustrated in FIG. 6.

In an alternative implementation, light from the phase modulator isfocused onto an array of integration rods. In this case the exact valueof the focal distance is of less importance. One may chose to focus alllight from each source region onto the input face of the integration rodarray, in other words f=D. As mentioned in that section, smallvariations in the focal distances for each region can be determined inorder to ensure similar light spread at the outputs of the array ofintegration rods.

The resulting lens per region may look something like that shown in FIG.5.

FIG. 6 shows the cross section of two lenses with different sizesfocusing light onto an SLM.

An example prototype implementation of SNS breaks down the target imageinto 8 y-zones by 16 x-zones (“x” being the horizontal width, and “y”being the vertical height of the image). The image is repeatedlybisected (alternating in the x, then y, then x directions) until thedesired number of regions has been obtained.

FIGS. 7A to 7C illustrate repeated bi-section for an input image ofMarilyn Monroe. The target image is 1080×1920 (HD-resolution), so eachof the 8×16 cells is 135×120 pixels large.

In FIG. 7B we see that the luminance requirement is 51 on the left handside of the image, and 48 on the right. As a result, the area of theleft hand side (LHS) will be 51/(51+48) of the total spatial phasemodulator area, and the right hand side (RHS) will be 48/(51+48) of thetotal spatial phase modulator area. Very little redirection will benecessary for this skew: only a small amount of added light should beincident on the LHS. Because of the left-leaning skew, the lenses thatwe form on the RHS should have a slight tilt or slope towards the left.

In FIG. 7C, the LHS and RHS of the image are further bisected. Thebisection of the LHS results in 55 for the top and 48 for the bottom.Therefore, the top left quadrant of this image will require more lightthan the bottom left quadrant. The tilt of the bottom lens will beslight in the upwards direction. This process is repeated for the RHSand so on with further bisections until the image is split into 8×16 subregions.

We now calculate the lens shape for each of these regions. The in-planefocal distances (x and y) for each of the 8×16 regions of the spatialphase modulator are determined according to Equation 2A. The tilt ofeach lens is determined by the central coordinates of the spatial phasemodulator regions and the corresponding display regions on the SLM; callthese points (x₁, y₁)_(i) for the i_(th) region on the spatial phasemodulator and (x₂, y₂)_(i) for the i^(th) display region on the SLM. Theshift m_(x,i) in x is then calculated by:mx _(i)=−(x _(2,i) −x _(1,i))/2f _(x,i)  [3A]and a similar expression can be used for the slopes in the y-direction.The lens shapes for each of the 8×16 regions are calculated using thederived focal distances and tilts inserted into Equation 1A.

An example resulting lens array for the fully bisected 8×16 zone SNSprocess is shown in FIGS. 8 and 9.

FIG. 10 shows the calculated result of bouncing light off the lens imageshown in FIG. 9. In this example, the distance between the spatial phasemodulator and the SLM was 170 mm.

Steered light from the spatial phase modulator shown in FIG. 10 can nowbe relayed onto the SLM. Note the units on the right hand side of FIG.10. We see that by redirecting the light using only 8×16 zones, we canreach peak luminance levels over 45× above that which a uniformlyilluminated imaging device could deliver. In some embodiments the peakluminance levels are in excess of 30 times or 40 times the full screenwhite level of the projector.

It is entirely possible that steering of light enables light levelsabove what is required for the image in question. In this case, a globalreduction in light source power can be implemented (for example bypulse-width modulating it), or some of the light can be steered into alight-dump or some of the light can be removed by focusing the lightthrough a variable aperture.

The regions where the various lenses meet (e.g. along edges of thesource regions) may be smoothed out in order to eliminate sharp edgesand possibly unwanted artifacts. A simple low-pass filter may suffice,or the offsets between neighbouring regions can be minimized for anadequate effect.

Approach 2: Equally Sized LETO Regions; Differently Sized SLM Regions.

For this discussion, we assume that a uniform light distribution isincident on the spatial phase modulator plane and is redirected onto atarget image plane at some distance from the modulator plane. Otherincoming light distributions can be used and accounted for. An SLM mayoptionally be placed in the target plane, but it is not important forthe immediately following discussion.

Uniform light incident on the modulator plane is redirected onto atarget image plane at some distance from the modulator plane. Themodulator plane is divided into segments (source regions) of equal area,each responsible for redirecting light onto a particular segment(display region) of the target image plane (see FIG. 11).

It is the intention of this approach that the portion of optical powerconfined in each display region relative to the overall target image isthe same as the portion confined in each source region relative to thatof the overall modulator.

The geometries of the display regions are subject to the desired imageplane illumination profile, and may be computed using algorithms such asthe Median Cut algorithm. In the Median Cut example, a target imageplane segment with one quarter of the optical power of the entire imagecan be achieved by redirecting light from a modulator plane segment withone quarter of the area of the overall modulator.

Phase Modulator

A phase profile established on the modulator is used to redirect lightto the target image plane in order to achieve the desired illuminationprofile. The phase profile can be computed on a source region-by-sourceregion basis, where light incident on a source region is redirected bythe phase profile on that source region toward the corresponding displayregion.

Calculating the phase profile for each source region can be made easierby defining both the source regions in the modulator plane and thedisplay regions in the target plane as a grid of points denoting therespective location and orientation of each region.

A typical choice for the number of points in a source region ormodulator segment is the number of pixels in that segment available forphase modulation. Each pair of corresponding source region and displayregion should have the same number of points distributed uniformlyacross the respective regions such that a one-to-one point map relatingeach pair of source region and display region is possible.

Redirection

Given the point map relating a particular pair of a source region and acorresponding display region, the phase profile that will achieve thedesired light redirection can be obtained by a number of differentapproaches. The relationship between the phase profile and surfaceprofile is given by the Hyugens-Fresnel principle. The gradient of thephase profile determines the steering effect of the phase profile onlight. The phase profile is related to the surface profile of a physicallens by the refractive index of the medium (for a physical lens) and thegoverning equations of wave optics.

Since it is well known that a phase profile can be related to a surfaceprofile of optical path lengths, the following approaches are describedin terms of path lengths rather than phase.

In one approach, the path length profile for the modulator segmentconsists of the physical distances separating corresponding points inthe segment pair, see FIG. 12.

Referring to FIG. 12, we see that the optical path lengths between themodulator plane point map and the corresponding target image plane pointmap can be expressed as:L _(i)=|{right arrow over (M _(i) T _(i))}|,  [4A]where M_(i) is the coordinate of a particular point i in the modulatorplane, T_(i) contains the coordinates of the corresponding point in thetarget image plane, and L_(i) is the length of the vector between thetwo points.

In other approaches, the center points of both a source region and acorresponding display region in the region pair are utilized. In one ofthose approaches, the path length profile consists of the distancesseparating the points in the source region (modulator segment) withpoints in a virtual plane located on the center of the display region(target plane segment) and that is normal to the vector connecting thesegment centers. The points in the virtual plane used for the distancescorrespond to positions where the virtual plane intersects linesconnecting the segment pair (see FIG. 13).

Referring to FIG. 13, the optical path lengths between the modulatorplane point map and the corresponding target image plane point map canbe expressed as:

$\begin{matrix}{{L_{i} = \frac{\overset{\rightarrow}{M_{c}T_{c}}\mspace{11mu}\bullet\mspace{11mu}\overset{\rightarrow}{M_{i}T_{c}}}{\overset{\rightarrow}{M_{c}T_{c}}\mspace{11mu}\bullet\mspace{11mu}\left( {\overset{\rightarrow}{M_{i}T_{i}}/{\overset{\rightarrow}{M_{i}T_{i}}}} \right)}},} & \left\lbrack {5A} \right\rbrack\end{matrix}$where {right arrow over (M_(c)T_(c))} is the vector connecting thesegment pair centers, and {right arrow over (M_(i)T_(c))} is the vectorconnecting point M_(i) on the modulator plane segment with T_(c), thecenter point of the corresponding target plane segment. The dot ●between the vectors denote the commonly used symbol for the vectordot-product.

In another approach, the path length profile consists of the distancesseparating the points in the source region (modulator segment) frompoints in a virtual parabola centered on the center of the correspondingdisplay region (target plane segment). The points in the virtualparabola used for the distances may be located where the linesconnecting the segment pair intersect at 90 degree angles with linesconnecting the virtual parabola points to the center of the target planesegment (see FIG. 14).

Referring to FIG. 14, the optical path lengths between the modulatorplane point map and the corresponding target image plane point map canbe expressed as:

$\begin{matrix}{{L_{i} = {\overset{\rightarrow}{M_{i}T_{c}}\mspace{11mu}\bullet\mspace{11mu}\left( {\overset{\rightarrow}{M_{i}T_{i}}/{\overset{\rightarrow}{M_{i}T_{i}}}} \right)}},} & \left\lbrack {6A} \right\rbrack\end{matrix}$

Another aspect of this invention provides other example methods fordetermining a configuration of a configurable optical element (e.g. arefractive or phase-modulating element) which will result in generationof a desired light field when light from a source interacts with theconfigurable optical element. In some embodiment the configurableoptical element is dynamically reconfigurable. Such methods may be usedto generate light fields corresponding to image data for high-dynamicrange projection. In some embodiments the image data comprises videodata and displaying frames of the video data comprises configuring theconfigurable optical element. In some alternative embodiments the methodis applied to define a configuration for a fixed physical lens (e.g. aconfiguration that can be applied to make a lens by molding, machiningetc.) in order to provide a lens that will create a desired image byinteracting with light from a light source.

FIG. 15 shows an example arrangement 10 of a generalized refractiveoptical element 12 interacting with light 14 from a light source 16.This arrangement represents a general projector. Arrangement 10 issimplified in that element 12 has a planar rear surface 12A and lightfrom light source 14 is collimated and arrives perpendicular to rearsurface 12A. These conditions are not absolutely necessary but theysimplify calculations and are useful to provide a clear explanation ofthe algorithms that may be applied to determine a configuration forelement 12. Algorithms suitable for generating a configuration forelement 12 may be modified in ways that will be apparent to those ofskill in the art for cases in which the optical system in which element10 is more complicated.

In arrangement 10, light reaches lens plane 12A travelling parallel tooptical axis 13, enter a physical lens 12 at a surface 12A perpendicularto optical axis 13 and is refracted on reaching the far surface 12B ofelement 12 after which the light travels to an image surface. Under theassumption that the thickness of element 12 can be neglected for mostpurposes (“thin lens assumption”) and surface 12B has relatively shallowgradients, the transmission coefficient of element 12 is near-constant.

The Fresnel equations serve as the imaging model when physical lenssurfaces are desired. These equations relate the incident andtransmitted angles (θ₁ & θ₂) to the refractive indices of the twomaterials (n₁ & n₂). Angles θ₁ & θ₂ are measured with respect to thesurface normal vector, N which points from material of element 12 tomaterial surrounding element 12. The incident and transmitted angles arerelated by the Snell equation:

$\frac{\sin\;\theta_{1}}{\sin\;\theta_{2}} = \frac{n_{2}}{n_{1}}$where an incident ray 14 is parallel to optical axis 13, normal N isconsequently oriented at θ₁ with respect to axis 13. The angle of thetransmitted ray θ_(t) with respect to optical axis 13 is thenθ_(t)=θ₂−θ₁. This results in the following expression for the angle θ₁required to yield a given angle θ_(t).

$\theta_{1} = {\tan^{- 1}\left( \frac{\sin\;\theta_{t}}{\frac{n_{1}}{n_{2}} - {\cos\;\theta_{t}}} \right)}$

For thin lenses that are aligned with the optical axis, a paraxialapproximation can be used which assumes sin θ≈θ and cos θ≈1. With thisassumption, the previous equation simplifies to:

$\theta_{1} = {\left( \frac{n_{2}}{n_{1} - n_{2}} \right)\theta_{t}}$where element 12 is replaced by a phase modulator the relationshipsimplifies to the following:θ₁=θ_(t)

These relationships determine how incoming light rays are deflected byeither physical refraction at the lens surface or by a phase-modulator.The goal of free-form lensing is to use these relationships to determinea lens or phase surface that focuses light in bright regions of thetarget image and defocuses light in dark regions. The following sectionsdiscuss three approaches that may be used to generate configurations forrefractive and/or reflective and/or phase shifting elements that willyield desired light fields when illuminated.

Approach 3: Paraxial Deblurring Formulation

The paraxial deblurring formulation couples the mapping of light fromsource to target with lens-surface computation by introducing a paraxialassumption to the image formation model that greatly simplifies theproblem to be solved.

The benefit of this approach is that the re-distribution of light isguaranteed to produce a valid physical lens that does not rely onappropriately chosen discrete elements. The challenge is that theproblem to be solved is a poorly conditioned biharmonic system thattends to converge slowly with iterative methods while being too dense tofactor and solve effectively, particularly on highly parallel hardwaresuch as GPUs or FPGAs.

This section introduces an alternative solver based upon deconvolution.Conditioning issues with the system are reduced by solving the problemas a deconvolution problem in Fourier space, resulting in speedups ofseveral orders of magnitude. The following sections introduce a basicparaxial model and then present an alternative formulation that issolved in Fourier space.

Image Formation Model

The image of a point on the lens plane on an image plane located atfocal distance f is given by the following equations for a physical lensphase surface, respectively.

$v^{*} = {v + {f\;{\tan\left( {\frac{n_{1} - n_{2}}{n_{2}}\theta_{1}} \right)}}}$v^(*) = v + f tan (θ₁)

These equations can be approximated with the following linear equationsusing the paraxial assumption that sin θ≈θ and cos θ≈1:

$v^{*} = {v + {f\frac{n_{1} - n_{2}}{n_{2}}\theta_{1}}}$v^(*) = v + f θ₁

Using the paraxial approximation, the angle θ₁ can further be related tothe gradient of the lens surface or phase surface p(v) giving:

$v^{*} = {v + {f\frac{n_{1} - n_{2}}{n_{2}}\text{∇}{p(v)}}}$v^(*) = v + f∇p(v)

By defining a nominal focal length {circumflex over (f)} to be f for aphase surface or

$\frac{n_{1} - n_{2}}{n_{2}}$f for a physical lens, these two formulas can be collapsed into a singleexpression. The determinant of the Jacobian, J, of this mapping fromv→v* then determines the magnification at any point on the image plane.

$\begin{matrix}{{J} = {\begin{bmatrix}{\frac{\partial\;}{\partial x}{v^{*}(v)}} \\{\frac{\partial\;}{\partial y}{v^{*}(v)}}\end{bmatrix}}} \\{= {{\frac{\partial\;}{\partial x}{v^{*}(v)} \times \frac{\partial^{*}\;}{\partial y}(v)}}} \\{= {{\begin{bmatrix}{1 + {\frac{\partial\;}{\partial x}{p(v)}}} \\{\frac{\partial\;}{\partial y}{p(v)}}\end{bmatrix} \times \begin{bmatrix}{\frac{\partial\;}{\partial x}{p(v)}} \\{1 + {\frac{\partial\;}{\partial y}{p(v)}}}\end{bmatrix}}}} \\{= {{\frac{\partial\;}{\partial x}\left( {v + {\hat{f}\text{∇}{p(v)}}} \right) \times \frac{\partial\;}{\partial y}\left( {v + {\hat{f}\text{∇}{p(v)}}} \right)}}} \\{= {{1 + {\hat{f}\frac{\partial^{2}\;}{\partial x^{2}}{p(v)}} + {\hat{f}\frac{\partial^{2}\;}{\partial y^{2}}{p(v)}}} = {1 + {\hat{f}\text{∇}^{2}{p(v)}}}}}\end{matrix}$

The magnification is inversely proportional to the brightness on theimage plane. Using the mapping v→v* and the above expression relates theintensity of the image of a point v, i.e.:

${I\left( {v + {\hat{f}{\nabla{p(v)}}}} \right)} = {\frac{1}{J} = \frac{1}{1 + {\hat{f}{\nabla^{2}{p(v)}}}}}$

This can subsequently be linearized via a first-order Taylor series toobtain the non-linear image-formation model in Equation 1.I(v+{circumflex over (f)}∇p(v))≈1−{circumflex over (f)}∇ ² p(v)  (1)

This image formation model can be expressed as an inverse problem inwhich a phase/lens surface is sought that reproduces a target image asclosely as possible. The resulting optimization problem is shown inEquation 2.p(v)*=argmin_(p(v))∫_(Ω)(I(v+{circumflex over (f)}∇p(v))−1+{circumflexover (f)}∇ ² p(v))² dΩ  (2)

In Equation (2), I(.) is the image data (intensity at each point in animage); p(v) is phase as a function of position v on the opticalelement; Ω is the area of the image; {circumflex over (f)} is thenominal focal length (defined above) and p(v)* is the solutionconfiguration for the optical element.

A function (v)* that minimizes Equation (2) defines the lens or phasesurface that best approximates the target image.

Solution Algorithm

This objective function provided by Equation (2) is non-linear due tothe term I(v+{circumflex over (f)}∇p(v)), which can be interpreted as awarping of the target image I. In order to obtain a linear model, alinearization of this warping may be introduced. Equation 2 may then beminimized in an iterative fashion as shown in Algorithm 1.

Algorithm1 1 Linearized optimization of Equation (2) Procedure PARAXIALCAUSTICS (I,f) // Initialize phase surface as a constant value: p₀ (v) ←0 // Initialize iteration counter and start solve: k ← 0 while k <k_(max) do: // Warp target image by current solution: I_(p) ^(k)(v) ← I(v + {circumflex over (f)}∇p_(k)(v)) // Update the current solution bysolving Equation (2)        p^(k+1)(v) ← argmin_(p(v)) ∫_(Ω) ( I_(k)^(p)(v) − 1 + {circumflex over (f)}∇²p(v))²dΩ // Update iteration index:k ← k + 1 // Return computed mapping: return p_(kmax)( v) // Stop

At each iteration of Algorithm 1, after discretization into pixels, alinearized least-squares problem is solved to minimize the sum ofsquared residuals ½∥I_(p) ^(k)−1+{circumflex over (f)}∇²p∥₂ ². Thisproblem can be solved using commercially available solvers and othersolvers currently known in the art. Algorithm 1 has been validated insimulations and on a physical prototype setup and produces good results.However, the problem is poorly conditioned due to the squaring of theLaplace operator ∇². For this reason, convergence using iterativesolvers can be slow, while the system density makes direct solversmemory intensive.

Approach 4: Solution in Fourier Domain

For periodic boundary conditions, the problem exemplified by Equation(2) can be solved even more efficiently in Fourier-space. One approachis to apply proximal operators. For an arbitrary convex function, F(x),the proximal operator, prox_(γF), (defined in Equation 3) acts like asingle step of a trust region optimization in which a value of x issought that reduces F but does not stray too far from the input argumentq.

$\begin{matrix}{{{prox}_{\gamma\; F}(q)} = {{\underset{x}{argmin}{F(x)}} + {\frac{\gamma}{2}{{x - q}}_{2}^{2}}}} & (3)\end{matrix}$

For a least-squares objective F(x)=½∥Ax−b∥₂ ², the resulting proximaloperator is shown below.prox_(γF)(q)=(γ+A ^(T) A)⁻¹(γq+A ^(T) b)

With periodic boundary conditions and A is a circulant matrix, this canbe evaluated extremely efficiently in Fourier-space, shown in Equation4.

$\begin{matrix}{{{prox}_{\gamma\; F}(q)} = {\mathcal{F}^{- 1}\left( \frac{{{\mathcal{F}(b)}{\mathcal{F}(A)}^{*}} + {{\gamma\mathcal{F}}(q)}}{{\left( {1 + \alpha} \right){\mathcal{F}(A)}^{2}} + \gamma} \right)}} & (4)\end{matrix}$

The notation

&

⁻¹ indicate the forward and inverse Fourier transform, * indicatesconvex conjugation and multiplication/division are performed pointwise.The parameter α>0 acts as an L₂ regularization parameter on the lenscurvature. L₂ gradient penalties were also tried but found to have anadverse effect on solution quality.

By defining A={circumflex over (f)}∇² and b=1−I_(p) ^(k)(v) andq=p^(k)(v), the problem can be solved iteratively in Fourier space,resulting in Algorithm 2.

Algorithm 2 Paraxial caustics in Fourier space procedurePARAXIALCAUSTICSFOURIER (I,{circumflex over (f)},γ)  //Initialize phasesurface as a constant value  p⁰(v) ← 0  //Initialize iteration counterand constant parameters  A ← {circumflex over (f)}∇²  k ← 0  while k <k_(max) do   //Warp target image by current solution   I_(p) ^(k)(v) ←I(v + {circumflex over (f)}∇p^(k)(v))   //initialize right hand side ofleast-squares problem   b ← 1 − I(v + {circumflex over (f)}∇p^(k)(v))  //Update the current solution by evaluating   //the proximal operatorin Equation 4   p^(k+1)(v) = prox_(γF)(p^(k)(v))   //update iterationindex   k ← k + 1  //Return computed mapping return p^(k) _(max) (v)

By caching the Fourier transform of p^(k)(v), Algorithm 2 can beimplemented with one image warping, some vector operations and oneforward/reverse Fourier transform per iteration. All of these operationsare highly parallelizable, either into per-pixel or per-scanlineoperations.

As shown, Algorithm 2 is a non-linear variant of a common proximalalgorithm, the proximal-point method, which is a fixed-point algorithmfor minimizing an arbitrary convex F consisting of recursively callingprox_(γF) by evaluating: x^(k+1)←prox_(γF)(x^(k)).

A difficulty in the deblurring formulation is in assigning boundaryconditions to the resulting lens/phase surface. It is desirable to map arectangular lens to a rectangular image area, however the periodicityassumption when using Fourier can result in severe distortions nearboundaries. FIG. 16A is an example image in which such distortions canbe seen, especially along central portions of the top and bottom imageboundaries.

Results

A selection of results for physical lenses made according to solutionsobtained using Algorithm 2 are shown in FIGS. 20A, 21A, 22A, 23A and24A. All lenses were computed at a resolution of 256×128 with a pixelpitch of 0.5 mm, a 100 mm focal length, with γ=1000 and α=2.0 usingmirrored padding. Non-uniform rescaling, due to non-power-of-two inputdimensions, resulted in a slightly wrong focal length. All renderingswere computed at 130 mm focal length using Blender+LuxRender with normalsmoothing and loop subdivision. All images are gamma corrected fordisplay with a gamma of 2.2. The border around each image shows nominalfull-screen white values. Computation times were approximately 1 secondper image, but there is substantial room for code optimization viaparallelization, pipelining and porting to GPU.

Algorithm 2 is able to reproduce relatively fine details. Redistributionof light is limited to roughly ¼ of the screen dimension, which canlimit contrast for some very high contrast images. Lowering thesmoothness parameter α can improve this, but might introduce artefactsas can be seen by comparing FIGS. 17B, 17C and 17D.

Approach 5: Area-Based Parameterization Formulation

Another approach to determining mappings from source to target isarea-based parameterization. Area-based parameterization methods arebased on subdividing the lens or phase surface into patches or regionswhich are then mapped onto the image-plane. Some examples of thisapproach are described for light field projectors in U.S. 61/893,270(Light Field Projectors and Methods) and U.S. 62/031,250 (Methods andApparatus for Light Steering Using Phase Modulated Imaging) both ofwhich are hereby incorporated herein by reference for all purposes.

Mappings from source to target may be embodied in Fresnel mappings inthe case of a physical lens or as gradients of the phase function in thecase of phase modulation. Regardless of which image formation model isused, a method must be provided to determine what region on the lensplane should map to a particular corresponding region in the image planefor best reproduction of a desired light pattern.

The intensity of light within a region in the image plane may becontrolled by varying the size of the corresponding region in the lensplane. Increasing the size of the corresponding region in the lens planewill increase the light intensity in the corresponding region of theimage plane.

One way to establish mappings between a lens plane and an image plane isto divide both the lens plane and image plane into regions havingboundaries and to establish a correspondence between regions of the lensplane and corresponding regions of the image plane. For example, FIG. 40shows schematically a lens plane 42 divided into areas 42A by boundaries42B and an image plane 44 divided into areas 44A by boundaries 44B. Asindicated by arrows 45, each area 44A of image plane 44 corresponds to acorresponding area 42A of lens plane 42.

At this point it is worth noting that it is convenient but not mandatorythat the image plane and lens plane are planar. In general, either orboth of these surfaces may be curved. Also, although it is the case insome embodiments, it is not mandatory that there be a 1:1 correspondencebetween regions 42A and regions 44A. For example, in some embodimentstwo or more regions 42A may correspond to one region 44A. Also, it isnot mandatory (although it is generally desirable) that regions 42Acompletely cover lens plane 42.

Conveniently, regions 42A tile lens plane 42 and regions 44A tile imageplane 44. Regions 42 may be called “source regions” and regions 44 maybe called “target regions” because regions 42 serve as sources of thelight that illuminate corresponding regions 44 to replicate a targetlight pattern.

Conveniently, boundaries 42A are parameterized such that the sizes ofregions 42A may be varied by altering the parameters that defineboundaries 42B. Boundaries 42A and 42B comprise straight lines in someembodiments. In other embodiments boundaries 42A and/or 42B are curved.

One way to define regions 42A and 44A is by a triangulation withpiecewise linear boundaries defining triangular regions. In suchembodiments, the boundaries of the triangles may be conveniently defined(parameterized) by positions of the triangle vertices. Triangle vertexdisplacements then correspond to gradients of the phase function, whileregions interior to the triangles correspond to areas of constantcurvature. With this area parameterization, the mappings map piecewiseconstant regions on the lens plane to piecewise constant regions in theimage plane.

An algorithm may be applied to find boundary configurations forboundaries 42B that will result in reproduction of a target lightintensity in areas 44A in the image plane. For example, to determinetriangle vertex point positions in the lens plane that will reproduce atarget intensity within each triangle in the image plane. Where the lensplane is uniformly illuminated by a light source, the light intensitywithin a region of the image plane is given by the ratio of areas of theregion of the image plane to the corresponding region(s) in the lensplane. In the following example, uniform illumination of the lens planeis assumed. However, the algorithm may readily be modified to accountfor non-uniformities in the illumination of the lens plane.

Example Embodiment

Input to the algorithm is a triangular mesh M={T, V}. Here V={V₁, . . ., V_(n)} is a set of vertices where v_(i)∈

² and T={T₁, . . . , T_(m)} where t_(j)∈

³ are integer indices into V defining oriented triangles. The collectionof triangles defines a piecewise linear discretization of spaceφ(x)={φ₁(x), . . . , φ_(m)(x)}. The signed area of t_(j) is thenA(V,t_(j))=−½(v_(t) _(j,2) −v_(t) _(j,1) )×(v_(t) _(j,3) −v_(t) _(j,1)).

The parameterization formulation of light redistribution seeks a set ofvertex positions V*={V₁*, . . . , V_(n)*} on a source surface such thatA(V*,t_(j))=I_(j)A(V,t_(j))∀j∈[1,m], where I_(j) is the target intensitywith respect to the source intensity. This source intensity is assumedconstant. It is straightforward to accommodate a known non-constantlight intensity from the light source. In some embodiments the sourcemay be controlled to provide a non-constant light intensity thatfacilitates display of a particular image. For example, the source maybe controlled to provide an intensity distribution that is more intensein regions corresponding to larger intensity in the image and lessintense in regions corresponding to darker regions in the image.

Since the target intensities may have wide variation, this condition canbe expressed by the following objective function:

$\begin{matrix}{{E_{T}\left( V^{*} \right)} = {\frac{1}{2}{\sum\limits_{j = 1}^{m}\left( {\frac{A\left( {V^{*},t_{j}} \right)}{I_{j} + ɛ} - {A\left( {V,t_{j}} \right)}} \right)^{2}}}} & (5)\end{matrix}$

Normalizing by the target intensity ensures that errors are weightedequally regardless of whether they correspond to bright or dark regionsof the target image. The constant 0<ε«1 serves to regularize the problemin the event that the target intensity is exactly zero.

Conservation of energy requires that Σ_(j=1) ^(m)A(V*,T)=Σ_(j=1)^(m)A(V,T) (assuming no losses in whatever optical system takes lightfrom the lens plane to the image plane). It is therefore desirable toadjust the total amount of light that reaches the image plane to matchthe integrated target intensity. This can be achieved by pre-scaling thesource intensity, for example, by modulating the output of a lightsource, passing light from the light source through an optical systemcomprising a variable aperture and/or including a light modulator in anoptical path between the lens plane and image plane.

A further physical constraint is that light cannot be subtracted.Therefore, negative-area source regions do not make physical sense. Analgorithm may include the constraint that A(V*,T_(j))≥0∀j, which alsorequires that the resulting parameterizations are bijective.

Vertex positions, V*, which result in a minimization of Equation 5 yieldtriangles corresponding to high target intensity levels dilating andtriangles corresponding to low target intensity levels contracting.There is little need for additional constraint on the vertex positionsbeyond that the resulting triangles maintain positive area.

Attempting to optimize Equation 5 directly can result in poorlydistributed vertices. An example is shown in FIGS. 18A and 18C. Accuracyof reproduction of a target light field (e.g. an image) can be improvedsignificantly by introducing curl-regularization, which restricts thesolution space to those with low curl. For example, compare FIGS. 18Band 18D to FIGS. 18A and 18C.

Approach 6: Adding Curl & Smoothness Regularization to Approach 5

An example curl-regularizer is defined by Equation 6, which is expressedper-triangle of the computational mesh.

$\begin{matrix}{{E_{\nabla \times}\left( V^{*} \right)} = {\sum\limits_{j = 1}^{m}{\int_{x \in t_{j}}^{\;}{\left( {\nabla{\times {\Psi_{j}\left( {{V^{*} - V},x} \right)}}} \right)^{2}{dx}}}}} & (6)\end{matrix}$

If the input is in the form of a tessellated regular grid, theregularizer can be equivalently expressed in the form of finitedifferences on the grid rather than its component triangles.

Incorporating curl-regularization results in lower distortion in thepoint mappings. Reducing curl in the point mappings also advantageouslyresults in vertex displacements that can be better approximated by thegradient of a smooth and continuous lens surface. This is because thepositions of the vertices are ultimately applied to define the gradientof the resulting lens or phase field, either explicitly in the case ofphase, or implicitly through the Fresnel mapping in the case of aphysical lens.

In addition to the curl-regularization, some embodiments also apply asmoothness regularizer. Equation 7 provides one example of a smoothnessregularizer.

$\begin{matrix}{{E_{\nabla}\left( V^{*} \right)} = {\sum\limits_{j = 1}^{m}{\int_{x \in t_{j}}^{\;}{\left( {\nabla{\Psi_{j}\left( V^{*} \right)}} \right)^{2}{dx}}}}} & (7)\end{matrix}$

An example optimization incorporating both curl and smoothnessregularizer terms is shown in Equation 8.V*=argmin_(V) *E _(T)(V*)+βE _(∇×)(V*)+αE _(∇)(V*)subject to:A(V*,T _(j))≥0∀j  (8)

Equation 8 is a non-convex quartic function of the vertex positions V*subject to quadratic constraints and is consequently non-trivial tooptimize. The following section describes approaches that may be used tofind optimal solutions to Equation 8.

Numerical Solution

In some embodiments the curl-regularized objective in Equation 8 issolved using the limited memory Broyden-Fletcher-Goldfarb-Shanno method(L-BFGS). Various implementations of L-BFGS are publicly available.These include libBFGS for the C programming language.

L-BFGS uses a history of objective function gradient evaluations tobuild an approximation to the inverse Hessian matrix to compute a searchdirection. Once found, a secondary 1D optimization is performed alongthis search direction seeking an approximate minimizer. AdvantageouslyL-BFGS does not require that the Hessian be re-evaluated for every valueof V*.

The non-negativity constraints A(V*,t_(j))≥0, prevent precomputingsystem matrices or preconditioners. These constraints may be implementedusing a log-barrier method which introduces a penalty term for eachtriangle. An example penalty term is shown in Equation 9 which may beadded to Equation 8.

$\begin{matrix}{{E_{N}\left( {V^{*},\mu} \right)} = {\sum\limits_{j = 1}^{m}{{- \frac{1}{\mu}}{\log\left( {A\left( {V^{*},t_{j}} \right)} \right)}}}} & (9)\end{matrix}$

Initially the barrier parameter μ is set to a large value that isprogressively decreased. For example, the barrier parameter may bescaled by a factor τ∈(0,1). The factor may be constant. The resultingpenalty rapidly becomes a more accurate approximation to the originalconstraint condition A(V*,t_(j))≥0.

Algorithm 3 Numerical optimization of area-based parameterization procedure AREAPARAMETERIZATIONSOLVE (I, {tilde over (V)}*, T)  //Initialize barrier parameter   μ ← 1   //Initialize initial mapping  V* ← {tilde over (V)}*   while μ > μ_(min)do    //Solve for updatedmapping via L-BFGS    $\left. V^{*}\leftarrow{{\arg\mspace{11mu}{\min_{V^{*}}{E_{T}\left( V^{*} \right)}}} + {\frac{\beta}{2}E_{\nabla} \times \left( V^{*} \right)} + {\frac{\alpha}{2}{E_{\nabla}\left( V^{*} \right)}} + {\delta\;{E_{N}\left( {V^{*},\mu} \right)}}} \right.$   //Adjust barrier parameter    μ ← τμ   //Return computed mapping  return V*

In many cases, the penalty can be omitted completely (e.g. by settingδ=0) since the inverse scaling by target intensity causes flippedtriangles to only occur in dark areas of the image. This dramaticallyimproves the performance of the method, since multiple optimizations atdifferent δ values can be replaced by a single optimization.

Solution in a Scale-Space

Although the curl-regularizer helps to restrict solution to those thatare integrable, since the objective function is a quartic function ofpoint mappings, it is possible for the optimization to become stuck in alocal minimum of the objective function in Equation 8. In order to helpimprove this, the optimization can be performed in a scale-space fromcoarse to fine.

In order to help avoid getting stuck in local minima, Equation 8 issolved in a scale-space from coarse to fine. Pseudo-code for this isshown in Algorithm 4.

Algorithm 4 Scale-space optimization of area-based parameterization  procedure AREAPARAMETERIZATIONRECURSIVE (I, k)    w ← width(I)/2^(k)   h ← height(I)/2^(k)    I_(C) ← BlurAndDownsample(I, w, h, σ)    if w< 10 or h < 10 then     //Base level, compute root parameterization     $\left. \sigma\leftarrow\frac{2^{k}}{2} \right.$     //Generate auniform triangulation of the image domain     V, T ← UniformMesh (w, h)    //Optimize for the mapped point positions     V* ←AreaParameterizationSolve (I_(C), V, T)     //Return computed mappings    return V*, T    else     //Recursively compute parameterization andlinearly upsample     V_(C)*, T_(C) ← AreaParameterizationRecursive (I,k + 1)     {tilde over (V)}*, T ← UpsampleLinear2X(V_(C)*, T_(C))    //Solve for current scale using {tilde over (V)}*, T as initialconditions     V* ← AreaParameterizationSolve (I_(C), {tilde over (V)}*,T)     //Return computed mappings     return V*, T

Provided that β≠0, Algorithm 4 ensures that the resultingparameterizations are bijective. This is guaranteed since triangles areupsampled by a factor of 2×, so every subdivided triangle isentirely-contained within a single source triangle.

The multiscale procedure allows the method to recover point mappingswith large displacement and low curl. This results in point displacementfields that are almost integrable. This may be addressed by the codewhen integrating the point displacements to compute the final lenssurface or phase function.

Phase & Lens Surface Generation

Once the parameterization is complete one can generate a physical lenssurface from the point displacements V−V*, where V represents the pointson the target image plane and V* represents the points on the lenssurface. These displacements determine the in-plane offset from a pointon the lens surface to the image plane and consequently determine theangle from the lens surface to the mapped point with respect to theoptical axis.

$\theta_{1} = {\tan^{- 1}\left( \frac{v - v^{*}}{f} \right)}$$\theta_{1} = {\frac{n_{2}}{n_{1} - n_{2}}{\tan^{- 1}\left( \frac{v - v^{*}}{f} \right)}}$

These formulas assume that incident light is parallel to the opticalaxis and are measured with respect to the optical axis in a planeparallel to the plane containing the optical axis and outgoing raydirection.

The normal of the phase/lens surface is consequently constrained to aplane parallel to the plane containing the optical axis and outgoing raydirection, making an angle with respect to the optical axis of θ₁.Integrating these normals, in the ideal case of curl-free displacements,yields the desired phase/lens surface. However, these vectors are onlydefined at mesh vertices. To accommodate this, the integration may beperformed using an unstructured mesh (e.g. using the finite elementmethod) or the normals may be resampled to the pixels of the phase/lenssurface. The following example implementation takes the latter approach.This allows flexibility in the integration method chosen.

To perform the resampling, the triangulation normals may be rasterizedonto an image representing the phase/lens surface. Phong interpolationmay be used in this rasterization which results in normal fields thatcan be exactly represented with piecewise quadratic patches.

If the resampled normal field is curl-free, the lens/phase surface canbe integrated directly by solving a Poisson equation. In practice theresampled normal field is usually not curl-free. This does notnecessarily imply that a physical lens cannot reproduce the targetnormal field, only that a continuous and smooth physical lens cannot.Non-smooth, and possibly even discontinuous, lenses can reproduce a muchwider range of normal fields, at the possible expense of visualartefacts near the discontinuities.

This leads naturally to the idea of using sparse optimization methods toperform the integration, seeking a lens surface that satisfies thenormal fields well except at a sparse set of kinks or discontinuities.These methods are attractive since they automatically determine thetopology of any non-smooth regions. This is unlike using proscribedpatches.

Some suitable sparse optimization methods are variations of leastabsolute deviation (LAD) problems, which is defined below:p=argmin_(p) ∥Gp−N∥ ₁  (10)

In Equation 10, the matrix G represents the discrete gradient operator,p is the lens or phase surface to be recovered and N is the targetnormal field. Variations of the LAD problem include using a sparsernorm, e.g. the zero norm or a non-convex but still continuous norm.After experimenting with several options, a weighted LAD formulation,shown in Equation 11, was chosen for a prototype embodiment.p=argmin_(p) ∥WGp−WN∥ ₁  (11)

W is a diagonal weighting matrix that is used to favor certaindiscontinuity locations over others. With two rows in the gradientmatrix per pixel in the resulting normal field, the weight for theW_(2i,2i) and W_(2i+1,2i+1) may be set to:

$\sqrt{\frac{1.0}{\max\left( {ɛ,a_{i}} \right)}},$where a_(i) is the mapped area of pixel i. This weighting functionmagnifies normal errors in dark regions, which encourages the L₁optimization to place discontinuities there. Alternative weighting couldconsider smoothness of the parameterization. Equation 11 may be solvedusing any number of numerical methods for sparse reconstruction,including ADMM, Primal-Dual methods or Linear Programming formulations.

The area parameterization methods described herein can be parallelizedon a GPU or FPGA or other suitable hardware since these methods can beperformed using a matrix-free algorithm that relies on only gradientevaluations and simple vector operations as inputs to a L-BFGSoptimization. Gradient computation can be performed analytically inparallel per-pixel.

Methods as described herein may be optimized for faster processingand/or more accurate rendition of a target light pattern in variousways. For example, L-BFGS can parallelize across dot and outer products.Furthermore, tuning of line-search parameters in the L-BFGS algorithm,parallelizing gradient computation and/or avoiding temporaries as wellas optimizing for cache reads and removing temporaries may result insignificant speed increases in comparison to the prototype system usedto generate the example images shown in FIGS. 106B, 107B, 108B, 109B,110B, 121B, 122B, 123B, 124B and 125B.

By exploiting the multiscale structure, faster methods with betterparallelizability could be achieved by performing the area optimizationusing a variation of the method which parallelizes over independent setsof vertices.

Undesirable artefacts in dark regions may be reduced by altering thenormal integration procedure to work on the triangle mesh, rather than aresampled pixel grid as described above. Further improvement could beachieved by optimizing for the projection of each lens/modulator pixelonto the target image rather than each target image pixel onto thelens/modulator. This would reverse the roles of light and dark in theresulting optimization, possibly leading to artefacts in bright regions,but would avoid resampling. A disadvantage is that the resultingoptimization is likely to be less stable: minor variations in thepositioning of a modulator/lens pixel may result in very rapidlychanging intensities within the target image when optimizing forprojections onto the target image.

FIGS. 43A and 43B illustrate a non-limiting example method 100. Method100 incorporates both multi-scale processing performed by loop 101Awhich repeats for increasing resolution levels and a variable barrierparameter implemented by loop 101B which repeats for decreasing valuesof a barrier parameter. Alternative methods perform only one or neitherone of loops 101A and 101B.

At block 102 source regions are initialized to provide an initialmapping 103. Block 102 may, for example, comprise assigning locations tovertices defining triangular or otherwise-shaped source regions. Thenumber of source regions (as well as the number of corresponding displayregions) may vary with the current scale/resolution.

Image data 105 is used to construct an objective function 109. In theillustrated embodiment, the resolution of image data 105 is set at block106 which also sets a current value for a barrier parameter. Block 106may, for example, comprise downsampling or downsampling and filteringimage data 105 to yield sized image data 107. Block 108 uses the sizedimage data to generate an objective function 109. Objective function 109is supplied to block 110 which solves for an updated mapping 115. Block110 may implement a L-BFGS solver algorithm, for example. Block 110 mayinvoke one or more of a curl regularizer 112A, a smoothness regularizer112B and an area penalty 112C as described above.

In block 116 the barrier parameter is updated (for example by scalingthe current value for the barrier parameter or selecting a next one of aplurality of decreasing barrier parameters. Block 118 checks to seewhether the updated barrier parameter is below the smallest barrierparameter value to be used. If the updated barrier parameter value isnot below the smallest barrier parameter value to be used (NO result inblock 18) processing loops back via loop 101B to repeat for the nextbarrier parameter value. Blocks 116 and 118 may be reversed in orderwith an appropriate change to the test of block 118.

In the case of a YES result at block 118 (indicating that all barrierparameter values for the current scale have been completed), block 120checks to see if all scales have been completed (e.g. to see whether thecurrent resolution is full resolution or a maximum resolution. If so,processing continues in FIG. 43B. Otherwise block 122 increases theresolution of mapping 115 and the objective function and processingloops back by loop 101A to obtain an updated mapping at the newresolution.

FIG. 43B illustrates a continuation of method 100 in which mapping 115is processed to drive a light projector. In block 124, the normals forsource areas defined by in mapping 115 are resampled to yield resamplednormal 117. In block 136 the resampled normal are integrated to yield atarget phase surface 119. In block 128 the phase shift values in phasesurface 119 are adjusted modulo 2π to yield phase modulatorconfiguration data 129 so that they are within the range of a phasemodulator. In block 130 the phase modulator is driven according to phasemodulator configuration data 129. In block 132 an image is displayed.

Approach 7: Assignment Problem Formulation

A variation of the above approach generates mappings from illuminationsource to target image using an assignment problem formulation insteadof or in addition to a formulation exemplified by Equation 8. Assignmentproblems and techniques for solving them are common within the field ofoperations research. An example definition of an assignment problem isshown in Equation 12 for a set of source points s_(i) and target pointst_(j) of equal cardinality.

$\begin{matrix}{{w = {\underset{w}{argmin}{\sum\limits_{i}{\sum\limits_{j}{{C\left( {i,j} \right)}w_{i,j}}}}}}{{{subject}\mspace{14mu}{to}\text{:}\mspace{14mu}{\sum\limits_{j}w_{i,j}}} = {1\mspace{11mu}{\forall i}}}} & (12)\end{matrix}$

The matrix C(i,j) is a cost function indicating the cost of mappingw_(i,j) units of source point i to target point j, while the constraintsensure that sources and targets are completely mapped. In the standardlinear assignment problem, the weights w_(i,j) are allowed to befractional. Variations can require binary w₁₁.

If s_(i) is a source position and t_(j) is a target position, commoncost functions C(i,j) are the Manhattan and Euclidean distances. In manycases, the cost function is sparse, meaning that only a subset ofpossible assignments (i,j) are permitted, with infeasible matchesimplicitly assigned infinite cost.

This problem can be applied to caustic generation by generating sourceand target point distributions proportionally to source and targetluminance and then computing the optimal assignments between source andtarget by solving Equation 12. These assignments then determine theoutgoing angles from the source, and the Snell mapping and normalintegration methods discussed above can then be used to arrive at aconfiguration for an optical element at the lens plane. To solveEquation 12 several approaches can be used including linear programming,or, in the discrete case seeking a 1:1 mapping, the auction algorithm orthe Hungarian algorithm.

Caustic formation via Equation 12 may be advantageous, e.g. to minimizeshear which induces curl in the resulting normal fields. Theseadvantages can be traded off against the computation expense of solvingEquation 12 on the point set in question. Equation 12 can requireconsiderable computation to solve especially for large point sets withnon-sparse cost functions. Introducing sparsity in distance (Manhattanor Euclidean) cost functions limits the steering effect of lensing,effectively constraining modifications to local regions.

In an example method, the assignment problem formulation is applied torefine the point mappings computed by the area parameterization methoddescribed above to reduce curl in the resulting normal maps. This wouldalso avoid having to solve a dense assignment problem, which iscomputationally expensive, replacing it instead with a sparse problemthat is quicker to solve.

Comparison of Results

This section presents a comparison of the paraxial deblurring and areaparameterization approaches. For paraxial-deblurring, all lenses werecomputed at a resolution of 256×128 with a pixel pitch of 0.5 mm, a 100mm focal length, with γ=1000 and α=2.0 using mirrored padding.Non-uniform rescaling, due to non-power-of-two input dimensions,resulted in a slightly wrong focal length. All renderings were computedat 130 mm focal length. Computation times were approximately 1 secondper image but there is substantial room for code optimization viaparallelization, pipelining and porting to GPU.

For area-parameterization, all lenses were computed with ε=0.05, α=0.05and β=10.0 and δ=0. This selection of parameters disables therequirement that all source areas (e.g. triangles) are constrained tohave positive area. However, the resulting parameterizations are oftenbijective or close to bijective. This can be seen in FIGS. 105A and105B, where even though dark regions of the image are severelycompressed (indicating very little light being mapped to them), theyremain convex. The tendency of the algorithm to yield bijective mappingsresults at least in part from the normalization by target intensity,which results in the optimization penalizing the relative error inachieving a target magnification factor rather than an absolute error.Computation time was approximately 5-10 seconds per frame, but could besped up by a factor of 4 by working at the same resolution as theparaxial deblurring results, bringing the two computation times closertogether.

A comparison of the Paraxial-deblurring and area-parameterizationapproaches is presented in FIGS. 20A to 24C. Each image has a greyborder that indicates the nominal incident illumination and a thin blackborder indicating no illumination. All images are gamma corrected with agamma of 2.2 except for the Einstein area parameterization image of FIG.20B which uses a gamma of 3.0. The images were rendered from mesh filesusing Blender+LuxRender with normal smoothing and Loop subdivisionenabled. Computation times were approximately 1 second per image, butthere is substantial room for code optimization via parallelization,pipelining and porting to GPU.

Overall, it can be seen that the paraxial-deblurring formulation does abetter job of reproducing fine details than the area-parameterizationmethod: results were computed at approximately ⅛ scale for theparaxial-deblurring method compared to ¼ resolution for thearea-parameterization and yet still show finer details than are presentin the area-parameterization results. This difference can be attributedmostly to the fact that the area-parameterization method can provideconsiderably stronger steering than the paraxial-deblurring method.

Both methods distort the input images somewhat. This is partly due toresizing artefacts and part due to the lenses becoming thick enough thatthe thin-lens assumption does not apply well. Some lenses have thicknessapproximately 10% of their focal lengths. Much of the distortion can becorrected by slightly adjusting the distance of the lens from theimage-plane.

Experimental Results

Several phase patterns were computed for use on a prototype projector.The prototype projector uses a coherent 532 nm green laser source whichis expanded and relayed onto a 2π “LETO” phase modulator. The output ofthe phase modulator is then relayed to a projection lens and onto thescreen. Most patterns used default parameters from the previous section.However, the “candle” image used ε=0.1. The “candle” (FIG. 36A),“Einstein” (FIG. 32A) and “avengers” (FIG. 35A) images used L₂integration.

Comparison images between the paraxial deblurring and areaparameterization results as captured using the same camera settings(ISO800, 0.01 s, F20) are shown in FIGS. 29B to 32C. The paraxialdeblurring results have relatively low contrast which can make itchallenging to see the structure of the image. The area parameterizationresults have much better contrast, but are sensitive to alignment withinthe projector optics as well as to uniformity of the illumination of theLETO. Minor misalignments in the projector can result in severe localdistortion of the resulting images; several of the results show thesedistortions.

FIGS. 33A to 33H show similar comparisons for broadband illumination ofthe LETO from a low-power white LED with the output image focused on awhite business card using the same phase patterns as in FIGS. 29B to29C. Distortions are reduced somewhat, as are artefacts introduced bywrapping in the phase patterns, at the expense of a raised black leveland chromatic aberrations. In this case the area-parameterization stillout-performs the paraxial deblurring approach.

A set of results from the area-parameterization method using camerasettings that highlight contrast is shown in FIGS. 34A to 34D. Theseimages show that the area-parameterization method diverts significantportions of the light from dark regions to light regions.

The technology described herein may be implemented in various ways. FIG.41 shows apparatus 50 according to one embodiment of the invention.Apparatus 50 comprises a processor 54 that takes in image data 52 thatspecifies a desired light field (which may be a picture, light pattern,video frame etc.) and produces data specifying a lens or phase modulatorconfiguration. Data 56 may be used to control a 3D printer 57A to yielda printed physical lens or a milling machine or other machining center57B to yield a physical lens or a mold for a physical lens or a datastore 57C for later use. Data 56 may be supplied to a light projector57D comprising a controllable lens, mirror/phase modulator or the like.

FIG. 42 illustrates a light projector 60 that comprises a controller 62connected to receive data 56. Note that the functions of controller 62and processor 54 of apparatus 50 may be combined such that lightprojector 60 takes in and processes image data. In other embodimentslight projector 60 may work together with a pre-processor apparatus thatpartially performs the functions of apparatus 50. Light projector 60 maythen perform further steps to process the data.

Projector 60 includes a light source 64 and a phase modulator 66. Insome embodiments phase modulator 66 is replaced by a dynamicallyvariable mirror or dynamically variable lens. Phase modulator 66 iscontrolled by display controller 62 to adjust the phase of lightincident from light source 64 on a pixel-by-pixel basis to cause adesired image to be displayed at image 68. Image 68 may comprise asurface onto which light is projected from the front or rear forexample.

Projector 60 may include one or more mechanisms to adjust the averagebrightness of a projected image to match the image data. The illustratedembodiment includes an optional control signal 64A which varies lightoutput of light source 64. The illustrated embodiment also includes anoptional global modulator 65 such as a variable aperture in the lightpath between light source 64 and phase modulator 66. Global modulator 65is operable to controllably attenuate the light incident at phasemodulator 66. In another example embodiment phase modulator 66 isconfigured to direct some light away from image 68 in cases wheredirecting all light received from light source 64 to image 68 wouldresult in higher than desired average image luminance.

Projector 60 also includes an optional clean up stage 67. Clean up stage67 may comprise a spatial light modulator such as an LCD panel ordigital mirror device or the like that is capable of adjustingtransmission of light to image 66 on a pixel basis. Clean up stage 67may be used to adjust average luminance of projected image and also maybe used to correct for artifacts in the projected images.

Interpretation of Terms

Unless the context clearly requires otherwise, throughout thedescription and the

-   -   “comprise”, “comprising”, and the like are to be construed in an        inclusive sense, as opposed to an exclusive or exhaustive sense;        that is to say, in the sense of “including, but not limited to”;    -   “connected”, “coupled”, or any variant thereof, means any        connection or coupling, either direct or indirect, between two        or more elements; the coupling or connection between the        elements can be physical, logical, or a combination thereof;    -   “herein”, “above”, “below”, and words of similar import, when        used to describe this specification, shall refer to this        specification as a whole, and not to any particular portions of        this specification;    -   “or”, in reference to a list of two or more items, covers all of        the following interpretations of the word: any of the items in        the list, all of the items in the list, and any combination of        the items in the list;    -   the singular forms “a”, “an”, and “the” also include the meaning        of any appropriate plural forms.    -   Words that indicate directions such as “vertical”, “transverse”,        “horizontal”, “upward”, “downward”, “forward”, “backward”,        “inward”, “outward”, “vertical”, “transverse”, “left”, “right”,        “front”, “back”, “top”, “bottom”, “below”, “above”, “under”, and        the like, used in this description and any accompanying claims        (where present), depend on the specific orientation of the        apparatus described and illustrated. The subject matter        described herein may assume various alternative orientations.        Accordingly, these directional terms are not strictly defined        and should not be interpreted narrowly.

Embodiments of the invention may be implemented using specificallydesigned hardware, configurable hardware, programmable data processorsconfigured by the provision of software (which may optionally comprise“firmware”) capable of executing on the data processors, special purposecomputers or data processors that are specifically programmed,configured, or constructed to perform one or more steps in a method asexplained in detail herein and/or combinations of two or more of these.Software may include or consist of instructions for configuring aconfigurable logic device such as a FPGA to implement logic forexecuting a method. Examples of specifically designed hardware are:logic circuits, application-specific integrated circuits (“ASICs”),large scale integrated circuits (“LSIs”), very large scale integratedcircuits (“VLSIs”), and the like. Examples of configurable hardware are:one or more programmable logic devices such as programmable array logic(“PALs”), programmable logic arrays (“PLAs”), and field programmablegate arrays (“FPGAs”)). Examples of programmable data processors are:microprocessors, digital signal processors (“DSPs”), embeddedprocessors, graphics processors, math co-processors, general purposecomputers, server computers, cloud computers, mainframe computers,computer workstations, and the like. For example, one or more dataprocessors in a control circuit for a device may implement methods asdescribed herein by executing software instructions in a program memoryaccessible to the processors.

Processing may be centralized or distributed. Where processing isdistributed, information including software and/or data may be keptcentrally or distributed. Such information may be exchanged betweendifferent functional units by way of a communications network, such as aLocal Area Network (LAN), Wide Area Network (WAN), or the Internet,wired or wireless data links, electromagnetic signals, or other datacommunication channel.

For example, while processes or blocks are presented in a given order,alternative examples may perform routines having steps, or employsystems having blocks, in a different order, and some processes orblocks may be deleted, moved, added, subdivided, combined, and/ormodified to provide alternative or subcombinations. Each of theseprocesses or blocks may be implemented in a variety of different ways.Also, while processes or blocks are at times shown as being performed inseries, these processes or blocks may instead be performed in parallel,or may be performed at different times.

In addition, while elements are at times shown as being performedsequentially, they may instead be performed simultaneously or indifferent sequences. It is therefore intended that the following claimsare interpreted to include all such variations as are within theirintended scope.

Software and other modules may reside on servers, workstations, personalcomputers, tablet computers, image data encoders, image data decoders,video projectors, video processors, video editors, audio-visualreceivers, displays (such as televisions), digital cinema projectors,media players, multi-processor systems, microprocessor-based orprogrammable consumer electronics, network PCs, mini-computers,mainframe computers, and the like as well as other devices suitable forthe purposes described herein. Those skilled in the relevant art willappreciate that aspects of the system can be practised with othercommunications, data processing, or computer system configurations.

The invention may also be provided in the form of a program product. Theprogram product may comprise any non-transitory medium which carries aset of computer-readable instructions which, when executed by a dataprocessor, cause the data processor to execute a method of theinvention. Program products according to the invention may be in any ofa wide variety of forms. The program product may comprise, for example,non-transitory media such as magnetic data storage media includingfloppy diskettes, hard disk drives, optical data storage media includingCD ROMs, DVDs, electronic data storage media including ROMs, flash RAM,EPROMs, hardwired or preprogrammed chips (e.g., EEPROM semiconductorchips), nanotechnology memory, or the like. The computer-readablesignals on the program product may optionally be compressed orencrypted.

In some embodiments, the invention may be implemented in software. Forgreater clarity, “software” includes any instructions executed on aprocessor, and may include (but is not limited to) firmware, residentsoftware, microcode, and the like. Both processing hardware and softwaremay be centralized or distributed (or a combination thereof), in wholeor in part, as known to those skilled in the art. For example, softwareand other modules may be accessible via local memory, via a network, viaa browser or other application in a distributed computing context, orvia other means suitable for the purposes described above.

Where a component (e.g. a software module, processor, assembly, device,circuit, etc.) is referred to above, unless otherwise indicated,reference to that component (including a reference to a “means”) shouldbe interpreted as including as equivalents of that component anycomponent which performs the function of the described component (i.e.,that is functionally equivalent), including components which are notstructurally equivalent to the disclosed structure which performs thefunction in the illustrated exemplary embodiments of the invention.

The following are non-limiting enumerated example embodiments of thedisclosed invention.

-   1. A method for controlling a phase modulator to display an image    defined by image data, the method comprising:    -   defining a plurality of non-overlapping source regions on a        two-dimensional phase modulator and a plurality of display        regions at a display plane, each of the source regions having a        boundary and a source area and being associated with a        corresponding one of the display regions; each of the display        regions having a corresponding display region area;    -   based on the image data, assigning a target light intensity        value to each of a plurality of the display regions;    -   adjusting: a configuration for the source regions; or a        configuration for the display regions; or configurations for        both the source regions and the display regions such that ratios        of the display areas of the display regions to the source areas        of the corresponding source regions is proportional to a ratio        of source light intensity values for the source regions to the        target light intensity value assigned to the corresponding        display region;    -   generating a phase surface for each of the source areas, the        phase surface configured to redirect light incident on the        source area onto the corresponding display area; and    -   controlling the phase modulator to provide the phase surfaces        for the source regions and illuminating the source regions with        incident light according to the source intensity values.-   2. A method according to example aspect 1 (or any other example    aspect herein) comprising determining target source areas based on    the image data and adjusting the configuration for the source    regions by performing an optimization to determine configurations    for the boundaries of the source regions which best satisfy an    objective function which quantifies aggregate deviations of the    areas of the source regions from the target source areas    corresponding to the source regions.-   3. A method according to example aspect 2 (or any other example    aspect herein) wherein generating the phase surfaces comprises,    based on the configurations of the source region boundaries after    the optimization, determining a normal vector for each of the source    regions and integrating the normal vectors to yield a solution phase    function relating a phase of the phase modulator to position in two    dimensions.-   4. A method according to example aspect 2 (or any other example    aspect herein) wherein the source regions comprise non-overlapping    source tiles defined by lines extending between a plurality of    source vertices, each of the source vertices having a location and    wherein the display regions comprise non-overlapping display tiles    defined by lines extending between a plurality of display vertices.-   5. A method according to example aspect 4 (or any other example    aspect herein) wherein the source tiles and display tiles are    triangles.-   6. A method according to example aspect 4 or 5 (or any other example    aspect herein) wherein the optimization determines optimized    locations for the source vertices.-   7. A method according to example aspect 1 (or any other example    aspect herein) comprising adjusting the configuration for the source    regions by performing a median cut algorithm.-   8. A method according to example aspect 1 (or any other example    aspect herein) or example aspect 7 wherein generating the phase    surface for each of the source areas comprises generating the phase    surface corresponding to a parabolic lens.-   9. A method according to example aspect 8 (or any other example    aspect herein) comprising defining the parabolic lens by a pair of    focal lengths on orthogonal directions based on differences in size    of the source areas and corresponding display areas in the    orthogonal directions.-   10. A method according to example aspect 9 (or any other example    aspect herein) wherein defining the parabolic lens comprises    specifying slopes in the two orthogonal directions, the slopes based    on displacements of the source regions relative to the target    regions in the orthogonal directions.-   11. A method according to any one of example aspects 1 to 10 (or any    other example aspect herein) wherein generating the phase surface    comprises low-pass filtering.-   12. A method according to any one of example aspects 1 to 11 (or any    other example aspect herein) wherein generating the phase surface    comprises phase wrapping.-   13. A method according to any one of example aspects 1 to 12 (or any    other example aspect herein) wherein illuminating the source regions    comprises controlling an output of a light source based on the image    data.-   14. A method according to example aspect 13 (or any other example    aspect herein) comprising controlling the output of the light source    based on an average luminance of the image.-   15. A method according to example aspect 13 or 14 (or any other    example aspect herein) wherein controlling the output of the light    source comprises passing the output of the light source through a    variable aperture and controlling a size of the variable aperture.-   16. A method according to any one of example aspects 13 to 15 (or    any other example aspect herein) wherein controlling the output of    the light source comprises varying an intensity of the light source.-   17. A method according to any one of example aspects 1 to 16 (or any    other example aspect herein) wherein at least 95% of the light    redirected by each of source regions falls within the corresponding    display region.-   18. A method according to any one of example aspects 1 to 16 (or any    other example aspect herein) wherein the light redirected by each of    the source regions substantially fills the corresponding display    region.-   19. A method according to any one of example aspects 1 to 18 (or any    other example aspect herein) comprising passing light from the phase    modulator to the display regions by way of an array of integrating    rods.-   20. A method according to any one of example aspects 1 to 19 (or any    other example aspect herein) comprising amplitude modulating light    from the display regions.-   21. A method for controlling a phase modulator to display an image    defined by image data, the method comprising:    -   providing a model of a two-dimensional light source comprising a        plurality of non-overlapping source regions, each of the source        regions having a boundary, a corresponding source light        intensity value and a source area and being associated with a        corresponding display region of a display, each of the display        regions having a corresponding display area;    -   based on the image data, assigning a light intensity value to        each of the display regions;    -   setting a target source area for each of the source regions such        that a ratio of the target source area of the source region to        the display area of the corresponding display region is        proportional to a ratio of the light intensity value assigned to        the corresponding display region to the source light intensity        value for the source region;    -   performing an optimization to determine configurations for the        boundaries of the source regions which best satisfy an objective        function which quantifies aggregate deviations of the areas of        the source regions from the target source areas corresponding to        the source regions;    -   based on the configurations of the source region boundaries        after the optimization, determining a normal vector for each of        the source regions;    -   integrating the normal vectors to yield a solution phase        function relating a phase of the phase modulator to position in        two dimensions.-   22. A method according to example aspect 21 (or any other example    aspect herein) wherein the source regions comprise non-overlapping    source tiles defined by lines extending between a plurality of    source vertices, each of the source vertices having a location and    wherein the display regions comprise non-overlapping display tiles    defined by lines extending between a plurality of display vertices.-   23. A method according to example aspect 22 (or any other example    aspect herein) wherein the source tiles and display tiles are    triangles.-   24. A method according to example aspect 22 or 23 (or any other    example aspect herein) wherein the optimization determines optimized    locations for the source vertices.-   25. A method according to example aspect 22 or 23 (or any other    example aspect herein) wherein the normal vectors are located at the    source vertices.-   26. A method according to any one of example aspects 21 to 25 (or    any other example aspect herein) wherein determining the normal    vectors for the source vertices is based on in-plane displacements    of the source vertices relative to corresponding ones of the display    vertices.-   27. A method according to example aspect 22 (or any other example    aspect herein) wherein determining the normal vectors comprises    determining inverse tangents of the quotients of the displacements    and an optical distance between the light source and the display.-   28. A method according to any one of example aspects 21 to 27 (or    any other example aspect herein) comprising determining the source    light intensities based on an average intensity of the image data.-   29. A method according to example aspect 28 (or any other example    aspect herein) comprising controlling an illumination source    illuminating a phase modulator according to the source light    intensities.-   30. A method according to any one of example aspects 21 to 29 (or    any other example aspect herein) wherein the source light    intensities are equal.-   31. A method according to any one of example aspects 21 to 30 (or    any other example aspect herein) wherein optimizing comprises    including a cost for curl of the solution phase function.-   32. A method according to example aspect 31 (or any other example    aspect herein) wherein the cost for curl is determined according to    E_(∇×)(V*)=Σ_(j=1) ^(m)∫_(x∈t) _(j) (∇×Ψ_(j)(V*−V,x))²dx.-   33. A method according to any one of example aspects 21 to 32 (or    any other example aspect herein) wherein optimizing comprises    including a cost for non-smoothness of the solution phase function.-   34. A method according to example aspect 33 (or any other example    aspect herein) wherein the cost for non-smoothness of the solution    phase function is determined according to    E _(∇)(V*)=Σ_(j=1) ^(m)∫_(x∈t) _(j) (∇Ψ_(j)(V*))² dx.-   35. A method according to any one of example aspects 21 to 34 (or    any other example aspect herein) wherein the source regions are    triangles.-   36. A method according to any one of example aspects 21 to 35 (or    any other example aspect herein) wherein the display regions are    triangles.-   37. A method according to any one of example aspects 21 to 36 (or    any other example aspect herein) wherein optimizing comprises    applying a limited memory Broyden-Fletcher-Goldfarb-Shanno    algorithm.-   38. A method according to any one of example aspects 21 to 37 (or    any other example aspect herein) comprising performing the    optimization in a series of iterations at progressively finer scales    such that, in each iteration the number of source vertices and    display vertices is increased and the vertex positions for an    immediately previous iteration are used as starting configurations    for a current iteration.-   39. A method according to any one of example aspects 21 to 38 (or    any other example aspect herein) wherein integrating comprises    resampling the normal vectors to provide a resampled normal vector    for each pixel of the phase modulator.-   40. A method according to example aspect 39 (or any other example    aspect herein) wherein resampling comprises performing Phong    interpolation on the normal vectors.-   41. A method according to any one of example aspects 21 to 40 (or    any other example aspect herein) wherein integrating comprises    applying a sparse optimization method.-   42. A method according to example aspect 41 (or any other example    aspect herein) wherein the sparse optimization method comprises    finding the solution phase function that minimizes a difference    between a gradient of the solution phase function and a field of the    normal vectors.-   43. A method according to example aspect 42 (or any other example    aspect herein) wherein the difference is a weighted difference that    magnifies normal errors in dark regions of the image.-   44. A method according to any one of example aspects 21 to 43    comprising initializing the source regions and display regions to    uniform triangulations.-   45. A method according to any one of example aspects 21 to 44    comprising constraining the optimization to require all of the    source regions to have positive area.-   46. A method according to example aspect 45 (or any other example    aspect herein) wherein constraining the optimization comprises    including in the objective function a penalty term for each source    region (or any other example aspect herein) wherein the penalty term    is proportional to an area of the source region and has a sign    opposite to a term of the objective function that quantifies the    aggregate deviations of the areas of the source regions from the    target source areas and the method comprises successively reducing a    proportionality parameter in the penalty term in each of a plurality    of iterations wherein the positions for the vertices determined in    one of the plurality of iterations are used as an initial condition    for a next one of the plurality of iterations.-   47. A method according to any one of example aspects 21 (or any    other example aspect herein) wherein a luminance within at least one    of the display regions exceeds a full screen white level.-   48. A method according to example aspect 47 (or any other example    aspect herein) wherein a peak luminance exceeds 30 times a full    screen white level.-   49. A method according to any one of example aspects 21 to 48 (or    any other example aspect herein) comprising amplitude modulating    light incident on the phase modulator such that different ones of    the source regions are illuminated by light of different    intensities.-   50. A method according to any one of example aspects 21 to 48 (or    any other example aspect herein) comprising uniformly illuminating    the phase modulator.-   51. A method for generating a desired light pattern, the method    comprising:    -   establishing a correspondence between source regions on a phase        retarding modulator and corresponding display regions in an        image plane;    -   determining from image data desired optical power densities for        the display regions;    -   adjusting one or both of the source regions and the display        regions using the image data to achieve a distribution of power        densities in the display regions corresponding to the image        data; and    -   controlling the phase modulator to provide a pattern of phase        shifts operative to redistribute light from each of the source        regions of the imaging chip to a corresponding one of the        display regions by scaling and/or shifting light incident on the        source regions of the phase modulator.-   52. A method according to example aspect 51 (or any other example    aspect herein) comprising configuring the source regions to provide    lenses having focal lengths configured to provide the scaling.-   53. A method according to example aspect 52 (or any other example    aspect herein) wherein the lenses have different focal lengths in x-    and y-directions.-   54. A method according to example aspect 52 (or any other example    aspect herein) comprising configuring the lenses to include slopes    configured to provide the shifting.-   55. A method according to example aspect 54 (or any other example    aspect herein) comprising separately controlling the slopes in x-    and y-directions.-   56. A method according to any one of example aspects 51 to 55 (or    any other example aspect herein) wherein controlling the phase    modulator comprises phase wrapping the pattern of phase shifts.-   57. A method according to any one of example aspects 51 to 55 (or    any other example aspect herein) comprising varying the areas of the    source regions.-   58. A method according to any one of example aspects 51 to 57 (or    any other example aspect herein) comprising varying the areas of the    display regions.-   59. A method according to any one of example aspects 51 to 58 (or    any other example aspect herein) wherein the source regions are    rectangular.-   60. A method according to any one of example aspects 51 to 59 (or    any other example aspect herein) wherein the display regions are    rectangular.-   61. A method according to any one of example aspects 51 to 60 (or    any other example aspect herein) wherein the source regions are    triangular.-   62. A method according to any one of example aspects 51 to 60 (or    any other example aspect herein) wherein the display regions are    triangular.-   63. A method according to any one of example aspects 51 to 62 (or    any other example aspect herein) wherein ratios of areas of the    source regions to the corresponding display regions are at least    equal to a ratio of optical power density at the source region to a    maximum optical power density specified in the image data for the    corresponding display region.-   64. A method according to any one of example aspects 51 to 62 (or    any other example aspect herein) comprising clipping the image data    to yield clipped image data wherein ratios of areas of the source    regions to the corresponding display regions are at least equal to a    ratio of optical power density at the source region to a maximum    optical power density specified in the clipped image data for the    corresponding display region.-   65. A method according to any one of example aspects 51 to 62 (or    any other example aspect herein) wherein ratios of areas of the    source regions to the corresponding display regions are at least    equal to a ratio of optical power density at the source region to a    mean optical power density specified in the image data for the    corresponding display region.-   66. A method according to any one of example aspects 51 to 65 (or    any other example aspect herein) wherein the optical power density    within at least one of the display regions exceeds a full screen    white level.-   67. A method according to example aspect 66 (or any other example    aspect herein) wherein a luminance of at least one of the display    regions exceeds 40 times the full screen white level.-   68. A method according to example aspect 66 (or any other example    aspect herein) wherein a luminance of at least one of the display    regions exceeds 30 times the full screen white level.-   69. A method according to any one of example aspects 51 to 69 (or    any other example aspect herein) comprising spatially amplitude    modulating light incident on the phase modulator such that different    ones of the source regions are illuminated by light of different    intensities.-   70. A method according to any one of example aspects 51 to 69 (or    any other example aspect herein) comprising uniformly illuminating    the phase modulator.-   71. A method according to any one of example aspects 51 to 70 (or    any other example aspect herein) comprising homogenizing light that    has been redirected by the phase modulator.-   72. A method according to example aspect 71 (or any other example    aspect herein) wherein homogenizing the light comprises passing the    light through an array of integration rods.-   73. A method according to any one of example aspects 51 to 73 (or    any other example aspect herein) comprising calculating the pattern    of phase shifts for the phase modulator on a source region-by-source    region basis.-   74. A method according to any one of example aspects 51 to 73 (or    any other example aspect herein) comprising establishing a first    grid of points in one of the source regions and a second grid of    points in a display region corresponding to the source region such    that there is a 1 to 1 correspondence between the points of the    first and second grids of points, determining path lengths    corresponding to pairs of corresponding ones of the points in the    first and second grids of points and setting the a pattern of phase    shifts in the source region according to the path lengths.-   75. A method according to example aspect 74 (or any other example    aspect herein) wherein the path lengths extend perpendicular to a    plane associated with the display region.-   76. A method according to example aspect 74 (or any other example    aspect herein) wherein the path lengths extend perpendicular to a    parabolic surface associated with the display region.-   77. A method according to any one of example aspects 74 to 76 (or    any other example aspect herein) wherein the first grid of points    comprises one point for each pixel of the phase modulator within the    source region.-   78. A method according to example aspect 51 (or any other example    aspect herein) wherein adjusting one or both of the source regions    and the display regions comprises executing an optimization    algorithm to find boundaries for the source regions and/or the    corresponding display regions such that ratios of the areas of the    source regions to the corresponding display regions provide a best    match to target optical power densities for the source regions.-   79. A method according to example aspect 78 (or any other example    aspect herein) wherein the optimization algorithm comprises a cost    function term that penalizes curl in a field of points defining the    source regions.-   80. A method according to example aspect 78 or 79 (or any other    example aspect herein) wherein the optimization algorithm comprises    a cost function term that penalizes lack of smoothness of the    pattern of phase shifts.-   81. A method for generating a light pattern defined by image data,    the method comprising:    -   for each of a plurality of light source regions determining a        size and location for a corresponding display region;    -   controlling a phase modulator to emulate an array of lenses,        each of the lenses corresponding to one of the light source        regions and configuring the plurality of lenses to have focal        lengths and slopes such that light incident on each of the        plurality of lenses is redirected onto the corresponding display        region.-   82. A method according to example aspect 81 (or any other example    aspect herein) comprising setting the sizes of the display regions    such that ratios of the areas of the source regions to the areas of    the corresponding display regions are proportional to luminance of    the source regions to luminance specified by the image data for the    corresponding display region.-   83. A method according to example aspect 82 (or any other example    aspect herein) wherein determining sizes and locations for the    display regions comprises processing the image data to iteratively:    -   divide a part of the image into plural parts such that areas of        the plural parts decrease with increases in average luminance        specified by the image data for the plural parts.-   84. A method according to example aspect 83 (or any other example    aspect herein) wherein dividing the part of the image into plural    parts comprises dividing the part of the image into two parts.-   85. A method according to example aspect 83 or 84 (or any other    example aspect herein) wherein the parts are rectangular in outline.-   86. A method according to example aspect 81 (or any other example    aspect herein) wherein determining the sizes and locations for the    display regions comprises performing a plurality of iterations of a    median cut algorithm.-   87. A method according to any one of example aspects 81 to 86 (or    any other example aspect herein) wherein controlling the phase    modulator comprises generating a phase surface corresponding to the    array of lenses and low-pass filtering the phase surface.-   88. A method according to any one of example aspects 81 to 87 (or    any other example aspect herein) wherein configuring the lenses    comprises phase wrapping.-   89. A method according to any one of example aspects 81 to 87 (or    any other example aspect herein) comprising controlling an output of    the light source based on the image data.-   90. A method according to example aspect 88 (or any other example    aspect herein) comprising controlling the output of the light source    based on an average luminance of the light pattern.-   91. A method according to example aspect 88 or 89 (or any other    example aspect herein) wherein controlling the output of the light    source comprises passing the output of the light source through a    variable aperture and controlling a size of the variable aperture.-   92. A method according to any one of example aspects 89 to 91 (or    any other example aspect herein) wherein controlling the output of    the light source comprises varying an intensity of the light source.-   93. A method according to any one of example aspects 81 to 92 (or    any other example aspect herein) wherein the display regions are    non-overlapping.-   94. A method according to any one of example aspects 81 to 93 (or    any other example aspect herein) wherein at least 95% of the light    redirected by each of the lenses falls within the corresponding    display region.-   95. A method according to any one of example aspects 81 to 94 (or    any other example aspect herein) wherein the light redirected by    each of the lenses substantially fills the corresponding display    region.-   96. A method according to any one of example aspects 81 to 94 (or    any other example aspect herein) comprising redirecting the light    onto the corresponding display regions by way of an array of    integrating rods.-   97. A method according to any one of example aspects 81 to 96 (or    any other example aspect herein) comprising amplitude modulating    light from the display regions.-   98. A method according to example aspect 97 (or any other example    aspect herein) wherein amplitude modulating the light comprises    controlling pixels of a spatial light modulator located to interact    with the light.-   99. A program product comprising a non-transitory data storage    medium having recorded thereon computer-readable instructions which,    when executed by a data processor, cause the data processor to    execute a method according to any one of example aspects 1 to 98 (or    any other example aspect herein).-   100. A program product comprising a non-transitory data storage    medium having recorded thereon machine-readable instructions which,    when executed by a data processor, cause the data processor to    configure a programmable logic device to perform a method according    to any one of example aspects 1 to 98 (or any other example aspect    herein).-   101. A light projector comprising:    -   a free form lens illuminated by a light source; and    -   a controller connected to control a configuration of the free        form lens, the controller configured to:        -   associate pixels of the free form lens to a plurality of            source regions, each of the source regions corresponding to            a display region;        -   based on image data, adjust relative sizes of the source and            corresponding display regions; and        -   control the pixels within each source region to cause light            incident on the source region to illuminate the            corresponding display region.-   102. A projector according to example aspect 101 (or any other    example aspect herein) wherein the free form lens comprises a    spatial phase modulator and the controller is connected to control    phase retardations provided by pixels of the spatial phase    modulator.-   103. A projector according to example aspect 101 or 102 (or any    other example aspect herein) wherein the controller is configured to    control an optical power of light from the light source incident on    the free form lens in response to the image data.-   104. A projector according to example aspect 103 (or any other    example aspect herein) wherein the controller is operative to    control amplitudes and/or widths and/or duty cycle of power supplied    to the light source.-   105. A projector according to example aspect 103 or 104 (or any    other example aspect herein) wherein the controller is connected to    control an optical element operable to selectively direct a portion    of light emitted by the light source to a light dump.-   106. A projector according to any one of example aspects 103 to 105    (or any other example aspect herein) comprising a variable aperture    in an optical path between the light source and the free form lens    wherein the controller is operable to control an opening of the    aperture.-   107. A projector according to any one of example aspects 101 to 106    (or any other example aspect herein) comprising an upstream spatial    light modulator in an optical path between the light source and the    free form lens wherein the controller is connected to control the    upstream spatial light modulator to differently illuminate different    ones of the source regions.-   108. A projector according to any one of example aspects 101 to 107    (or any other example aspect herein) comprising a downstream spatial    light modulator located in an optical path downstream from the free    form lens, the controller connected to control pixels of the    downstream spatial light modulator to vary amplitudes of light in a    light pattern produced by the projector at a target plane.-   109. A projector according to example aspect 108 (or any other    example aspect herein) wherein the downstream spatial light    modulator has a resolution sufficient to provide a plurality of    pixels operable to modulate light from each of the display regions.-   110. A projector according to any one of example aspects 101 to 109    (or any other example aspect herein) comprising an array of    integration rods in an optical path between the free form lens and    the display regions wherein the controller is operable to control    the free form lens to selectively steer different amounts of light    into different ones of the integrating rods.-   111. A projector according to any one of example aspects 101 to 110    (or any other example aspect herein) wherein the controller    comprises a programmed data processor.-   112. A projector according to any one of example aspects 101 to 111    (or any other example aspect herein) wherein the controller    comprises a configurable logic unit and a data store comprising    instructions for configuring the configurable logic unit.-   113. A projector according to example aspect 112 (or any other    example aspect herein) wherein the configurable logic unit comprises    a FPGA.-   114. Apparatus for controlling a free form lens to display an image    defined by image data, the apparatus comprising a processor    configured by software instructions to:    -   define a plurality of non-overlapping source regions on a        two-dimensional phase modulator and a plurality of display        regions at a display plane, each of the source regions having a        boundary and a source area and being associated with a        corresponding one of the display regions; each of the display        regions having a corresponding display region area;    -   based on the image data, assign a target light intensity value        to each of a plurality of the display regions; and    -   determine: a configuration for the source regions; or a        configuration for the display regions; or configurations for        both the source regions and the display regions such that ratios        of the display areas of the display regions to the source areas        of the corresponding source regions is proportional to a ratio        of source light intensity values for the source regions to the        target light intensity value assigned to the corresponding        display region and the configuration causes light incident on a        source area to be redirected onto the corresponding display        area.-   115. Apparatus according to example aspect 114 (or any other example    aspect herein) comprising a driver circuit connectable to drive a    free form lens.-   116. Apparatus according to example aspect 114 or 115 (or any other    example aspect herein) wherein the free form lens comprises a    spatial phase modulator and the apparatus is configured to generate    a phase surface for each of the source areas.-   117. Apparatus according to any one of example aspects 114 to 116    (or any other example aspect herein) comprising an optimizer    configured to perform an optimization to determine configurations    for the boundaries of the source regions which best satisfy an    objective function which quantifies aggregate deviations of the    areas of the source regions from the target source areas    corresponding to the source regions.-   118. Apparatus according to example aspect 117 (or any other example    aspect herein) wherein the optimizer comprises a curl regularizer.-   119. Apparatus according to example aspect 117 or 118 (or any other    example aspect herein) wherein the optimizer comprises a smoothness    regularizer.-   120. Apparatus for controlling a phase modulator to display an image    defined by image data, the apparatus comprising:    -   a controller configured with a model of a two-dimensional light        source comprising a plurality of non-overlapping source regions,        each of the source regions having a boundary, a corresponding        source light intensity value and a source area and being        associated with a corresponding display region of a display,        each of the display regions having a corresponding display area;    -   the controller configured by software instructions to cause the        controller to:    -   based on the image data, assign a light intensity value to each        of the display regions;    -   set a target source area for each of the source regions such        that a ratio of the target source area of the source region to        the display area of the corresponding display region is        proportional to a ratio of the light intensity value assigned to        the corresponding display region to the source light intensity        value for the source region;    -   perform an optimization to determine configurations for the        boundaries of the source regions which best satisfy an objective        function which quantifies aggregate deviations of the areas of        the source regions from the target source areas corresponding to        the source regions;    -   based on the configurations of the source region boundaries        after the optimization, determine a normal vector for each of        the source regions; and    -   integrate the normal vectors to yield a solution phase function        relating a phase of the phase modulator to position in two        dimensions.-   121. Apparatus for generating a desired light pattern, the apparatus    comprising:    -   a light source;    -   a phase retarding modulator illuminated by the light source;    -   a controller configured to:        -   establish a correspondence between source regions on the            phase retarding modulator and corresponding display regions            in an image plane;        -   determine from image data desired optical power densities            for the display regions;        -   adjust one or both of the source regions and the display            regions using the image data to achieve a distribution of            power densities in the display regions corresponding to the            image data; and        -   control the phase modulator to provide a pattern of phase            shifts operative to redistribute light from each of the            source regions of the imaging chip to a corresponding one of            the display regions by scaling and/or shifting light            incident on the source regions of the phase modulator.-   122. Apparatus for generating a light pattern defined by image data,    the apparatus comprising:    -   a light source;    -   a phase modulator illuminated by the light source;    -   a controller configured to, for each of a plurality of light        source regions:    -   determine a size and location for a corresponding display        region; and    -   control the phase modulator to emulate an array of lenses, each        of the lenses corresponding to one of the light source regions        and configuring the plurality of lenses to have focal lengths        and slopes such that light incident on each of the plurality of        lenses is redirected onto the corresponding display region.-   123. A controller for a light projector comprising a data processor,    and a data store comprising computer-readable instructions for    execution by the data processor, the instructions configured to    cause the data processor to execute a method according to any one of    example aspects 1 to 99.-   124. A method for controlling a free form lens to display an image    defined by image data, the method comprising:    -   defining a plurality of non-overlapping source regions on the        free form lens and a plurality of display regions at a display        plane, each of the source regions having a boundary and a source        area and one or more source intensity values and being        associated with a corresponding one of the display regions; each        of the display regions having a corresponding display region        area;    -   based on the image data, assigning a target light intensity        value to each of a plurality of the display regions;    -   adjusting: a configuration for the source regions; or a        configuration for the display regions; or configurations for        both the source regions and the display regions such that ratios        of the display areas of the display regions to the source areas        of the corresponding source regions is proportional to a ratio        of source light intensity values for the source regions to the        target light intensity value assigned to the corresponding        display region;    -   generating a configuration for the free form lens in each of the        source areas, the configuration arranged to redirect light        incident on the source area onto the corresponding display area;        and    -   controlling the free form lens according to the configuration        and illuminating the source regions with incident light        according to the source intensity values.-   125. Apparatus having any new and inventive feature, combination of    features, or sub-combination of features as described herein.-   126. Methods having any new and inventive steps, acts, combination    of steps and/or acts or sub-combination of steps and/or acts as    described herein.

Specific examples of systems, methods and apparatus have been describedherein for purposes of illustration. These are only examples. Thetechnology provided herein can be applied to systems other than theexample systems described above. Many alterations, modifications,additions, omissions, and permutations are possible within the practiceof this invention. This invention includes variations on describedembodiments that would be apparent to the skilled addressee, includingvariations obtained by: replacing features, elements and/or acts withequivalent features, elements and/or acts; mixing and matching offeatures, elements and/or acts from different embodiments; combiningfeatures, elements and/or acts from embodiments as described herein withfeatures, elements and/or acts of other technology; and/or omittingcombining features, elements and/or acts from described embodiments.

It is therefore intended that the following appended claims and claimshereafter introduced are interpreted to include all such modifications,permutations, additions, omissions, and sub-combinations as mayreasonably be inferred. The scope of the claims should not be limited bythe preferred embodiments set forth in the examples, but should be giventhe broadest interpretation consistent with the description as a whole.

What is claimed is:
 1. A light projector comprising: a free form lensilluminated by a light source, the free form lens comprising atwo-dimensional array of pixels; and a controller connected to control aconfiguration of the free form lens, the controller configured to:associate the pixels of the free form lens to a plurality of sourceregions, each of the source regions corresponding to a corresponding oneof a plurality of display regions; based on image data defining animage, adjust relative sizes of the source regions and the correspondingdisplay regions; and control the pixels within each source region tocause light incident on the source region to illuminate thecorresponding display region.
 2. The projector according to claim 1wherein the free form lens comprises a spatial phase modulator and thecontroller is connected to control phase retardations provided by pixelsof the spatial phase modulator.
 3. The projector according to claim 2comprising an array of integration rods in an optical path between thefree form lens and the display regions wherein the controller isoperable to control the free form lens to selectively steer differentamounts of light into different ones of the integrating rods.
 4. Theprojector according to claim 2 wherein the controller is configured tocontrol an optical power of light from the light source incident on thefree form lens in response to the image data.
 5. The projector accordingto claim 1 wherein the controller is configured to control an opticalpower of light from the light source incident on the free form lens inresponse to the image data.
 6. The projector according to claim 1wherein there is a 1:1 correspondence between the source regions and thedisplay regions.
 7. The projector according to claim 1 wherein thesource regions are defined by source region boundaries, the displayregions are defined by display region boundaries and the source regionboundaries are parameterized, wherein adjusting the relative sizes ofthe source regions and the corresponding display regions comprisesaltering one or more parameters that define the source regionboundaries.
 8. The projector according to claim 7 wherein the sourceregion boundaries comprise piecewise linear boundaries definingtriangular source regions.
 9. The projector according to claim 1 whereinthe controller is operable to control the light source based on theimage data to provide a light intensity distribution incident on thefreeform lens that is more intense in those of the source regionscorresponding to larger intensity in the image and less intense in thoseof the source regions corresponding to darker regions in the image. 10.The projector according to claim 1 wherein the controller is configuredto determine a 1:1 mapping of source points in the source regions tocorresponding display points in the display regions.
 11. The projectoraccording to claim 1 wherein the controller is configured to control thephase modulator to emulate an array of lenses, each of the lensescorresponding to one of the light source regions and configuring theplurality of lenses to have focal lengths and slopes such that lightincident on each of the plurality of lenses is redirected onto thecorresponding one of the display regions.
 12. A light projectorcomprising: a free form lens illuminated by a light source, the freeform lens comprising a two-dimensional array of pixels; and a controllerconnected to control a configuration of the free form lens, thecontroller configured to: associate the pixels of the free form lens toa plurality of source regions, each of the source regions correspondingto a corresponding one of a plurality of display regions; based on imagedata defining an image, adjust relative sizes of the source regions andthe corresponding display regions; and control the pixels within eachsource region to cause light incident on the source region to illuminatethe corresponding display region; wherein the controller is operable toexecute a Broyden-Fletcher-Goldfarb-Shanno optimization algorithm tofind optimized locations for vertices defining boundaries for the sourceregions and/or the corresponding display regions such that ratios of theareas of the source regions to the corresponding display regions providea best match to target optical power densities for the display regions.13. The projector according to claim 12 wherein the optimizationalgorithm comprises an objective function comprising a cost functionterm that penalizes curl in a field of points defining the sourceregions.
 14. The projector according to claim 13 wherein the objectivefunction comprises a cost function term that penalizes lack ofsmoothness of the pattern of phase shifts.
 15. The projector accordingto claim 12 wherein the controller is further configured to refine thefreeform lens configuration by generating source and target pointdistributions proportionally to source and target luminance, anddetermining outgoing angles from the source points by computing optimalassignments between the source points and the target points.
 16. Theprojector according to claim 15 wherein the controller is furtherconfigured to determine the configuration of the freeform lens by Snellmapping and normal integration.
 17. The projector according to claim 15wherein the assignments comprise a 1:1 mapping of the source points tothe display points.
 18. A light projector comprising: a free form lensilluminated by a light source, the free form lens comprising atwo-dimensional array of pixels; and a controller connected to control aconfiguration of the free form lens, the controller configured to:associate the pixels of the free form lens to a plurality of sourceregions, each of the source regions corresponding to a corresponding oneof a plurality of display regions; based on image data defining animage, adjust relative sizes of the source regions and the correspondingdisplay regions; and control the pixels within each source region tocause light incident on the source region to illuminate thecorresponding display region; wherein the controller is configured todetermine a 1:1 mapping of source points in the source regions tocorresponding display points in the display regions and determine adesired configuration for the freeform lens by determining outgoingangles from the source points to the corresponding display points and toperform Snell mapping and normal integration using the determinedoutgoing angles.
 19. A light projector comprising: a free form lensilluminated by a light source, the free form lens comprising atwo-dimensional array of pixels; and a controller connected to control aconfiguration of the free form lens, the controller configured to:associate the pixels of the free form lens to a plurality of sourceregions, each of the source regions corresponding to a corresponding oneof a plurality of display regions; based on image data defining animage, adjust relative sizes of the source regions and the correspondingdisplay regions; and control the pixels within each source region tocause light incident on the source region to illuminate thecorresponding display region; wherein the controller is configured toestablish a first grid of points in one of the source regions and asecond grid of points in a display region corresponding to the sourceregion such that there is a 1 to 1 correspondence between the points ofthe first and second grids of points, determine path lengthscorresponding to pairs of corresponding ones of the points in the firstand second grids of points and setting the a pattern of phase shifts inthe source region according to the path lengths.
 20. A light projectorcomprising: a free form lens illuminated by a light source, the freeform lens comprising a two-dimensional array of pixels; and a controllerconnected to control a configuration of the free form lens, thecontroller configured to: associate the pixels of the free form lens toa plurality of source regions, each of the source regions correspondingto a corresponding one of a plurality of display regions; based on imagedata defining an image, adjust relative sizes of the source regions andthe corresponding display regions; and control the pixels within eachsource region to cause light incident on the source region to illuminatethe corresponding display region; wherein the controller is configuredto perform a method comprising: defining the plurality of source regionson the free form lens and the plurality of display regions at a displayplane, wherein the source regions are non- overlapping, each of thesource regions have a boundary and a source area and one or more sourceintensity values and being associated with a corresponding one of thedisplay regions and each of the display regions has a correspondingdisplay region area; based on the image data, assigning a target lightintensity value to each of a plurality of the display regions;adjusting: a configuration for the source regions; or a configurationfor the display regions; or configurations for both the source regionsand the display regions such that ratios of the display areas of thedisplay regions to the source areas of the corresponding source regionsis proportional to a ratio of source light intensity values for thesource regions to the target light intensity value assigned to thecorresponding display region; generating a configuration for the freeform lens in each of the source areas, the configuration arranged toredirect light incident on the source area onto the correspondingdisplay area; and controlling the free form lens according to theconfiguration and illuminating the source regions with incident lightaccording to the source intensity values.
 21. A light projectorcomprising: a free form lens illuminated by a light source, the freeform lens comprising a two-dimensional array of pixels; and a controllerconnected to control a configuration of the free form lens, thecontroller configured to: associate the pixels of the free form lens toa plurality of source regions, each of the source regions correspondingto a corresponding one of a plurality of display regions; based on imagedata defining an image, adjust relative sizes of the source regions andthe corresponding display regions; and control the pixels within eachsource region to cause light incident on the source region to illuminatethe corresponding display region; wherein the controller is configuredto numerically solve for optimized locations for vertices definingboundaries for the source regions and/or the corresponding displayregions such that ratios of the areas of the source regions to thecorresponding display regions provide a best match to target opticalpower densities for the display regions by minimizing an objectivefunction that comprises a sum of functions of differences between: aratio of the area of one of the source regions to a desiredcorresponding target intensity; and the area of a corresponding one ofthe display regions.
 22. The projector according to claim 21 wherein theobjective function comprises a cost function term that penalizes curl ina field of points defining the source regions.
 23. The projectoraccording to claim 21 wherein the objective function comprises a costfunction term that penalizes lack of smoothness of the pattern of phaseshifts.